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PREFACE 



Much effort has been directed toward the development of procedures and 
instruments for the evaluation of human characteristics and abilities. Fre- 
quently, for the purpose of predicting future achievement, tests have been 
developed to measure general achievement. 



The literature on general achievement tests is rather extensive. 

However, the evaluation of occupational competence seems to attract attention 
only during periods of emergencies. World War I brought on the Oral Test, 
developed by the U. S. Army for the purpose of more effective classification 
of millions of civilians for Army occupations. 



When the depression years created millions of unemployed, the United 
States Employment Service carried on extensive efforts to classify people 
according to certain skill levels. Oral tests were further developed. World 
War 11 brought additional refine ■ ^nts and a limited introduction of perfor- 
mance evaluation. 



The Civil Service has played a major role in developing testing programs 
for the selection of individuals for a wide variety of jobs. Many of the 
Civil Service tests are of the written type. But, in recent years, practical 
performance tests have been introduced. 



Similar ly , the military, faced with an urgent need for better marmower 
utilization, has carried occupational competency testing to a fr r 1 sup 1 
ticated level in selected and special occupationa. 



Vocational industrial- technical education, experiencing rapid expansion 
and the need for continuous qualitative improvements, is faced with the 
problem of selecting competent tradesmen and skilled technologists for 
teaching positions in the many varied vocational education programs. The 
need for valid and reliable measures for evaluating the competency of individ- 
uals from industry is urgent. 

Despite the great need for such measurement, the effort to develop reli- 
able instruments and techniques of measurement has received less attention 
than the need demands. Published information for test development is sparce 
Indeed . 
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This "Handbook for Developing and Administering Occupational Competency 
Tests” is an outgrowth of extensive study of occupational competing testing 
on a national scale. It represents a summary of the best existing practices 
and procedures employed by the Civil Service, the military, various State 
Education Departments, labor, industry, vocational industrial- technical 
teacher education institutions, and the life-time experiences of the authors. 



As the title implies, this "Handbook” is intended to assist in the devel- 
opment, administration and evaluation of written and performance tests wher- 
ever occupational competency evaluation may be fundamental to employment, 
upgrading or promotion. 



The material has been organized in four main parts: Part I provides the 

historical background and philosophical concepts of occupational competency 
testing. These relate to the selection of tradesmen and technicians from 
industry for teaching in vocational programs; the nature of written, perfor- 
mance and oral tests, and their advantages and disadvantages. 



Part II is concerned with test development procedures, the personnel 
involved in test construction and actual test construction. Examples are 
provided for illustrative purposes. The procedures involved in test adminis- 
tration are discussed in Part III. Part IV is devoted to the evaluation of 
occupational competency tests and test results. The findings concerning 
national occupational competency testing are reviewed in Part V. 



Appendices of significant resources, references and a bibliography com- 
plement this report. 



The contents o , particularly the recommendations, reflect 

a careful synthesizing of the best-tested experiences interpreted against 
many, many years of practical work in occupational competency testii-g by the 
project director am: codirector. The contents, therefore, do not cessar ily 
reflect the position or policy of either Rutgers University or the :ed 
States ( - 'ice of Education, and no official endorsement by either . lization 
should e inferred. 



New Brunswick, New Jersey 
February I, 1971 
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foreword 



Instruction and learning in vocational education are bridged by the appli- 
cation of skills and information to a particular job. Every time a student 
has satisfactorily completed a job, he has, in fact, completed a performance 
test. Evaluating the finished product by checking the corrected trouble in a 
piece of service equipment, the function of a replaced part, or che tuned-up 
engine, are steps practically automatic to the skilled tradesmen and relatively 

simple to perform. The measurement is specific and measurable- does the 

part function properly ---is it made according to specification ---are all 
dimensions within prescribed tolerances? 



It is more difficult to appraise such factors as: the correct procedure 
employed, the speed with which the job was done, and the manner in which tools, 
equipment and machines were handled. Although the finished product may meet 
all the specified requirements, there is no assurance that the learner fully 
understood the underlying principles involved and the technical knowledge re- 
quired. While it is assumed that he did, there may be no measurable evidence. 
The teacher foruis a judgement of the student's progress by asking questions, 
g ving assignments in writing, and ob^arving the learner in action. The type 
of work the student does indicates a level of manipulative proficiency and 
answers to questions provide a measure of understanding. 



The informational factors, combined with the instructor's understanding of 
the requirements of the trade, enable him to evaluate the learners achievement 
and proficiency. 



The key to effective instruction in vocational programs is the occupational 
competence of the instructor. How can such competency be evaluated? A number 
of procedures have been employed over the years and they consist basically of 
tour elements: 



1* appraisal of a teacher candidate's experience in the 
form of recommendations from employers, 

2. Oral questioning during an interview, 

3. written tests concerning the knowledge, information 
and judgements of the occupation, and 

4. actual performance under real or simulated working 
conditions . 



Whatever limitations these procedures contain, they are a step in the 
direction of establishing some objective evidence of occupational competence 
over purely subjective judgements. 
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In recent years, there has grown an increasing concern about accurate and 
reliable testing procedures which can provide helpful information for evalu- 
ating a prospective teacher's knowledge and skill of the trade which he ex- 
pects to Leach , and the occupational proficiency which he has acquired. 



This Handbook contains no magic formula for test planning, development, 
validation, nor is there a simple or inexpensive solution to the problem. 
Constructing a good test is a demanding, time-consuming task, challenging the 
best creative of fort of a tradesman, test specialist and the test administrator. 
However, a summary of current best thinking and practices in occupational test 
construction and administration, presented in convenient form for reference, 
should prove useful to individuals responsible for the selection of tradesmen 
for teaching, for certification, and for admission to undergraduate and 
graduate programs of study. 



Carl Schaefer 



New Brunswick, New Jersey 
February 1, 1971 
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Chapter I 



TYPES AND PURPOSES OF OCCUPATIONAL COMPETENCY TESTS 



I. PURPOSE OF TESTING ON A NATIONAL SCALE 
A. Concern for Reliable and Valid Occupational Competency Tests 

Vocational administrators and industrial teacher educators have been 
increasingly concerned for reliable measures to evaluate the occupational 
competence of prospective trade and industrial- technical teachers. 

A number of states — notably. New York, California, Pennsylvania and 
F ior ida--have had the successful passing of occupational competency exam- 
inations a requirement for certification. Other states have started, or 
are planning to start, using such examinations. A number of Industrial 
Teacher Education Institutions are using such tests for admission to and 
advanced standing in undergraduate and graduate programs. 

Many state vocational education officials have stated that theix^ 
departments are not equipped to develop new examinations or to keep cur- 
rent ones up to date. There is general agreement that the development, 
printing, distribution, administration, scoring and validation impose a 
heavy burden upon limited resources. No state reported it was satisfied 
with its occupational competency testing program. Further, even the most 
highly industrialized states indicated their limitations to satisfactor- 
ily carry on such a program alone. 

School administrators, when they employ a new trade or industrial- 
technical teacher, want to be assured that the candidate possesses the 
personal qualities which are necessary for success as a teacher. They 
are, also, concerned with the individual's professional preparation. 

But, these two essential prerequisites are of little value without evi- 
dence of competence in a trade or industrial- technical occupation which 
the individual is expected to teach. 

The level of competence of the tradesmen is the keystone upon which 
the whole instructional program is built. Thus, the administrator (and 
the industry whom the program serves) must have assurances that the range 
of occupational competenc of the instructor is adequate to meet the* full 
spectrum of youth and adult training needs. 



B. 




Constraints to Test De velopm ent * 

Few states have been able to develop or participate in establishing 
accurate and reliable instruments suitable for evaluating the occupa- 
tional competence of prospective trade and Indus trial- technical education 
teachers. These instruments must evaluate skills, related technology, 

so 
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judgements and other understandings, mastered and further developed under 
actual wage-earning conditions. The development, administration and val- 
idation of occupational competency tests require human and material re- 
sources and finances that are in short supply even in leading industrial 
states. While each state recognizes that instructional excellence must 
be rooted in the occupational competence of each teacher, accurate mea- 
sures for evaluating such excellence have not been develops to any 
practical degree. 



C , Dissatisfaction with Existing Instruments 

The need to establish a national coordinated effort for competency 
appraisal has been sounded in loud, continuous and clear voices, decade 
after decade. Then, in 1959, a survey by Dr. Carl Schaefer showed that 
sixteen states were using tests to evaluate trade competency. However, 
discussions with officials responsible for vocational education and 
teacher certification in these states revealed that considerable dissat- 
isfaction existed with the quality of these tests. 

In 1966, representatives of twenty- three states met at Rutgers 
University in a seminar to study the feasibility of providing trade com- 
petency examinations on a national base. The outcome of the seminar in- 
dicated that the development of occupational competency examinations 
nationwide would be a more efficient use of personnel and should provide 
a higher quality examination. Almost unanimous agreement was expressed 
-hat such examinations would be used by most of the states. 



D . Duplication of Effort 




In a recent survey of the state-of-the-art of occupational compe- 
tency testing, it was established that twenty centers, located in sixteen 
states, are engaged in the preparation and administration of competency 
tests It was also found that twenty centers produced tests in the same 
trade area; fifteen centers duplicated their efforts for another trade 
area, and ten centers created tests for a number of additional trades. 
Each duplicated the efforts of the other. The duplication ranged from 
three to twenty tests in occupations little affected by special local or 
geographic conditions. 

These tests were largely of the paper and pencil variety, although 
some require actual performance in simulated job situations. There is 
little evidence to indicate that test items were pretested or that the 
instruments were validated according to acceptable test development pro- 
cedures, Little attention seems to have been given to such important 
technical considerations as reliability of scores or the objectivity of 
scoring procedures. 

The survey furthermore revealed that most of the test development 
and administration is carried on by individuals with many other duties 
who are able to devote only a limited amount of time to this effort. At 
the present time, only three states have assigned one person each to a 
testing program. Many states express^d^^ieed for help with the prepara- 
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tion and administration of trade proficiency tests for a teacher testing 
program. 



Economy and Efficiency of a National Effort 

A national effort would reduce or eliminate much costly duplication 
now found among a number of states. Substantial economies should be 
achieved and the quality of the tests greatly improved. National effort, 
resulting in accurate and objective evidence of occupational competency, 
would provide flexibility in the use of such tests; all states would have 
access to each examination. Each state would retain wide latitude in the 
use and interpretation of the results. 

A national effort can meet the states' concern about the reliability 
of occupational tests and their own limited resources to establish the 
necessary evidence for occupational competency. A National Occupational 
Competency Testing Program would have a wholesome effect upon the voca- 
tional education teaching profession in a manner similar to the national 
tests for professional architects, accountants, and others. 



II. HISTORICAL BACKGROUND OF TESTING 
Quantitative Measurements 

"Whatever exists, exists in some amount" has served as a slogan for 
a vast amount of effort that has gone into the construction of measuring 
instruments for teachers who wish to make an accurate report of the pro- 
gress of their students. When it is further asserted that "What exists, 
exists in some amount and can be measured" the slogan becomes a goal 
toward which test constructors are striving. 

Measuring the amount of things is foundational to the worldwide 
scientific revolution of modern times. Precise measurement is one of the 
crucial elements of the scientific method of inquiry and work. The scien- 
tist observes and collects his facts with the aid of instruments of obser- 
vation and measurement, organizes his facts in quantitative frequencies, 
treats these measured facts with mathematical methods, qualifies his find- 
ings with statistical reliability and draws conclusions only from such 
validated data. 



Units of Measurements 

To measure what can be measured requires "scales of equal units" so 
that every aspect of the physical universe can be measured with precision; 
weight, time, temperature, circular measure, electrical power, etc. The 
natural sciences became actualities as measuring Instruments with scales 
of ®*l ua I units perfected. In the building of these sciences, it was 
assumed that these were true facts which man could approximate by improv- 
ing measuring instruments, methods of using the instruments, and the 
treatment of the measurements. Hie measuring movement in all the sciences 
devoted much energy to "cutjiggdown error" in the observed facts. 2 



c. 



Measuring of Human Characteristics 



Attempts at measuring human characteristics and school achievement 
date back well over one hundred years. Much early work was done in 
Western Europe, Germany and France, and later by Americans. Oscar Boros 
published his first Mental Measurements Yearbook in 1938. He listed 1181 
instruments of mental measurement scales, tests, questionnaires, personal 
history records, rating scales, score cards and other objective instru- 
ments which were available that year, and 814 books, monographs and bul- 
letins dealing with the movement. The number has substantially increased 
and descriptions have been published in subsequent yearbooks. 



D. Uncertainties of Performance Measurements 

In spite of the enormous amount of careful work measuring human 
characteristics and school achievement, measuring the performance of 
human beings still involves uncertainties. Much of the effort has been 
concentrated on acts of habit formulation in school skills: handwriting, 
arithmetic, linguistic, historical, mathematical and scientific facts and 
knowledge. Acts of thinking, reasoning and problem solving in the natu- 
ral and social sciences; acts of creativeness, expression and apprecia- 
tion applied to the work of the schools, are included in this massive 
testing effort. 



E. Written tests are Dominant 



Among the many forms of tests developed over the years, the written 
test dominates. The most common objective tests involve a number of 
different forms of these types of test items: multiple choice, completion, 
true-false questions, simple recall, matching, analogy, short essay, etc. 
The dominance of these type of test items is not surprising for the evalu- 
ation of student achievement has been the major concern of psychologists 
and educators alike. Paper and pencil tests, and test results, lend them- 
selves to more readily objective treatment; the data can be treated more 
efficiently and acceptable evidence established. 

While progress in the measurement of verbal skills, facts, and infor- 
mation; as well as, intelligence, the appraisal of personality traits, 
occupational interests, mechanical comprehension, spatial visionalization 
and abstract reasoning has been substantial, attempts at evaluating occu- 
pational skills and know-how by using written tests and performance tests 
have been relatively limited. 
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III. BASIC CONCEPTS AND CHARACTERISTICS OF DIFFERENT 
TYPES OF OCCUPATIONAL COMPETENCY TESTS 

A. General Concepts 

Guidelines for the development of occupational competency tests and 
their ultimate use as measuring instruments are not available. This 
Handbook contains material gathered from many sources not easily acces- 
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slble through the general media of publications. The guidelines are pre- 
sented as major criteria which must be applied to the development of occu- 
pational competency tests. The practicality of developing written and 
performance tests for a given occupation can be determined through a re- 
view of the criteria. These guidelines and criteria have been prepared 
primarily for the purpose of relating competency proficiency evaluation 
to the selection of people from industry for teaching in vocational edu- 
cation programs. 



B . What is an Occupational Competency Test? 

An occupational competency test requires an individual to perform 
job- like tasks required of a craftsman or skilled technician within an 
occupation according to industrial standards of speed, accuracy, proce- 
dures, etc. The test is administered under well-controlled conditions to 
measure the individual's job proficiency through his ability to perform 
actual or job- like tasks. A well-defined goal is essential. The goal is 
the result the examinee is trying to achieve as he performs in a test 
situation. The goal should be the same for everyone and examinees must 
try to achieve the same results. 

If one individual works for speed and the other for accuracy, the 
test does not measure what it should. The test must be conducted so as 
to give each examinee an equal chance to demonstrate his skill and occu- 
pational knowledge and to assure that the test conditions are the same 
for all examinees. 



C. What are Job- Like Tasks? 



Job- like tasks include those tasks, derived from an occupational 
analysis, which are representative of the on-the-job work performed by 
a competent individual in a particular occupation. For most occupations, 
these involve manipulative, as well as informational skills. The rela- 
tionship between manipulative and knowledge skills varies from occupa- 
tion to occupation, and requires careful analysis. It is important to 
test for both the manipulative proficiency and the technical knowledge, 
trade information, judgements, etc. essential to carrying out the work 
of the occupation at an acceptable competency level. 



D. Relationship of Performance Testing to Pencil and Paper Testing 

1. Performance tests are, generally, assumed to be superior to 
written t^sts in measuring the job behavior of tradesmen and technicians. 
However, they are time-consuming, expensive to develop and administer, 
and pose difficulties in scoring. While written tests are sometimes 
used in place of performance tests, they do not measure manipulative/ 
performance competency. 




2. For the measurement of occupational proficiency, it is essential 
to administer both performance and written tests. Some test specialists 
maintain that only performance tests are a measure of real occupational 
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proficiency . Others insist that some pencil and paper tests can be made 
job- like and are, also* performance tests. In occupational competency 
testing, neither of these points of view is accurate and applicable. 

While performance tests, in general, are more likely to be job- like than 
pencil and paper tests, certain aspects of information, facts, computa- 
tions and interpretations can be more effectively measured through writ- 
ten tests. The determining factor, as to which test or combination of 
tests to use, is essentially "What is to be measured" . 

3. Tests can have the appearance of being good tests, even when 
they are not useful for measurement purpose. The original intent of the 
test constructor is nearly always good. However, the final test which 
he produces may or may not be adequate from the measurement standpoint, 
depending on what sacrifices and compromises or judgements are made in 
the course of the test development. 

4. If practical considerations, such as, time, availability of ma- 
chines or equipment, etc., have a dominant influence in determining what 
the performance tests are like, all of the original purposes may not be 
represented in the final product. 

When time is a factor, the test constructor may provide directions 

on how to proceed which would enable the examinee to proceed faster. 

While this is permissible, it must not eliminate or reduce the opportun- 
ity to measure the intended skill. One of the major purposes of a test 
is to determine whether the examinee knows how to perform the task with- 
out being told how. If the examinee is told how to proceed, the test 
will only measure whether he can perform the task after he has been given 
much orientation. The test may then no longer measure what was origin- 
ally intended. The performance test should go beyond pencil and paper 

tests and should measure the actual doing. 



E. Content of Written Proficiency Tests 

Written tests are usually easier to administer, quicker to score and 
provide more objective results. They are effective instruments to ap- 
praise certain aspects of occupational competence. The content of writ- 
ten tests should include the following areas of occupational competence: 

1. The factual knowledge and technological (trade theory) 
information of the occupation. 

2. Knowledge and understanding of the machines, tools, 
instruments, and apparatus of the occupation. 

3* The properties and characteristics of materials and 

the parts, or systems, which are diagnosed, processed 
and replaced. 
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4. The necessary calculations required for setting up, 
adjusting, testing antf bperating the machinery, 
apparatus, test equipment used in the occupation. 
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5. The reading and interpretation of specifications, 
tecnnical drawings a wiring and circuit diagrams, and 
other graphic representations. 

6. The interpretation of scientific principles as they 
a PPly to the design and/or operation of machines, 
apparatus , instruments and the processing of materials, 
component parts and accessories. 

7. Judgements and procedures involved in planning the work 
of the occupation, selection of materials and/or the 
replacement of component parts. 

8* Classification of materials, component parts and specific 
tools, instruments and supplies. 

9* General knowledge and work habits of the occupation, as 
well as specific requirements relating to occupational 
safety and hygiene, government regulations, and other 
conditions surrounding the occupation. 



F . Types and Uses of Performance Proficiency Tests 

A performance test for occupational proficiency generally involves 
a work sample test which requires the examinee to demonstrate his acquired 
skill by doing an actual segment of work using tools, materials, machines 
and equipment characteristic of the occupation for which the test is de- 
signed. 

Although, for a long period of time, it was assumed by some that 
written tests provided valid information to establish a level of occupa- 
tional competence, recent observations and the results of studies have 
i n di- c «ifced that written tests, alone, do not provide adequate and reliable 
evidence for this purpose. The relationship between the ability to an- 
swer test items on an examination and to perform in an actual job situ- 
ation is very low. 3 The relation between verbal and mechanical intelli- 
gence has been shown by repeated use of tests to be very slight. 4 There- 
fore, it is essential that an applicant for teaching in trade and 
industrial- technical education demonstrate his job skill and the required 
job knowledge in a practical situation typical of the occupation. 

1- A good performance proficiency test : 

a. Involves the measurement of critical skills of the 
occupation which distinquish the competent craftsman 
from the less competent individual. 

b. Includes selected manipulative processes which avoid 
extended repetition. 
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2- Well-designed performance tests must include: 



a. The actual operation of machines, apparatus, instru- 
ments and tools as utilized in a job situation. 

b • Step-by-step procedures in designing, forming, shaping, 
turning, cutting materials into a finished product, or 
diagnosing and assembling units and components. 

c. Step-by-step procedures in locating difficulties in 
equipment, instruments and/or apparatus. 

d. The planning and actual replacement of defective 
parts or components. 

e. The adjusting, maintaining, and utilizing of instru- 
ments of various kinds in carrying on the work of the 
occupat ion. 



G Specific Types of Tests 



There are three broad categories of tests which involve active 
"doing" on the part of the individual being tested: 

• The Recognition Test 

• The Simulated Condition Test 

• The Work Sample Test 
1- Recognition Test 

The "Recognition Test" or "Identification Test", as the 
names imply, attempt to measure the individuals competence 
to recognize the essential characteristics of an operation, 
or the product of performance, or to identify objects by 
specific name, functional or operational value, etc. For 
example, an individual may be shown a series of splices of 
electrical wires and be asked to determine those that are 
correctly spliced and those that are incorrectly spliced. 

In other performance tests of the recognition type, the in- 
dividual may be asked to identify parts of an automotive 
engine and cneir function. In order to achieve a maximum 
score, a testee would have to identify the engine parts in 
a definite order. 




Value of Recognition Test 

a. A performance test, with recognition items, is rela- 
tively easy to construct and can be adopted to a fairly 
wide range of subjects or operations. Although the 
test items measure some aspects of performance, they 
do not measure directly individual's skill, 

technique or procedure.^ ■ 
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The recognition test: 

b. Provides a variety of activities suitable for testing 
candidates in both the general area of the occupation 
and specialized divisions, 

c. Can be conducted in a well-equipped school shop labora- 
tory or commercial establishment, 

d. Provides occupational activities which the average in- 
structor should be able to perform to a marked degree 
of proficiency, 

e. Must include a variety of assignments in order to 
determine the all-around ability of the candidate, 

f. Should provide assignments of different lengths of time 
for completion so that a variety of occupational prac- 
tices may be completed within the time limit established, 

g- Must include assignments which can be evaluated object- 
ively by a competent and unbiased examiner. Its ratings 
must be objective, must differentiate and must distin- 
guish between individuals who differ by a certain 
amount in trade ability. "The performance test signi- 
fies what is commonly meant by a man’s competence to 
follow his trade, occupation or profession. Any com- 
plex set of coordinations which are required in a de- 
finite order and which characterize all men skilled in 
a given trade, thereby segregating them as a homogeneous 
group’ 1 ^ ... is a further definition of occupational 

competence . 

2 . Simulation Tes t 

The simulation technique of imitating actual conditions 
was developed during World War II and is best represented 
by the link trainer which aided the Air Force in the train- 
ing of pilots. Simulation tests are designed to isolate 
and duplicate the essential activities of a task, work, job, 
or sequence of operations which are necessary to analyze 
and correct a malfunction of a piece of equipment. The 
simulation technique has been carried to a very high degree 
of proficiency in the training program of astronauts. 

In simulation tests, the candidates carry out realistic 
tasks in a setting which simulates a real situation. Simu- 
lation techniques have the advantage of economy of teaching/ 
testing time, convenience, ease of administration and ob- 
servation. For example, in the interest of safety, tests 
are often devised to demonstrate the condition under which 
a fatal electric shock will occur if the man on the job 
performs an operation incorrectly. Thus, in simulating 
work with high, power lines, high voltage equipment in 
electronica orj^pneumatic clamping devices, a red light or 
a bell would iggate that an error has been made which 
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might well cause serious injury or fatality if the opera- 
tion had been performed in this manner on an actual line. 

Although the use of simulated conditions and miniature 
tests have certain advantages in the safe measurement of 
performances, test constructors must be cautions in assum- 
ing the degree of relationship between simulated or minia- 
ture tests and the more complete performance with actual 
equipment and under realistic conditions. What may seem 
to be a valid sample of an isolated task or a series of 
steps, may not be a valid sample of the total job situation. 
Simulated conditions of test performance, therefore, must 
be recognized and overcome if they are to serve the pur- 
poses intended. 

3 . Work Sample Test 

A work sample test requires that the individual demon- 
strate his skill and work knowledge by doing a series of 
tasks, such as correcting difficulties in a piece of elect- 
ronic equipment or completing a piece of work under actual 
work conditions. This type of test is realistic in that 
it requires the individual to carry through the actual oper- 
ations that the job demands under normal working conditions. 



In practical use, the performance test may consist of 
selected samples of a job, a product or part to be com- 
pleted within a certain period of time; or the partial com- 
pletion of an assignment or product. For many types of 
occupations, a completed job is not economical. For example, 
a complete engine overhaul in auto trades would be too time 
consuming. Generally, the more limited the sample job 
(which will predict the whole job) the greater the advan- 
tage as far as economical test administration is concerned. 
Therefore, it is important to find selected samples of 
performance that are sufficiently indicative of the indiv- 
idual's occupational competence to insure adequate measure- 
ment of his proficiency. ® 

Work sample tests have greater real-life appearances or 
face validity than any other type of test. When work 
sample tests are administered under standard conditions 
with standardized scoring procedures which have been care- 
fully developed, they can provide valid and reliable esti- 
mates of job performance. 



Uses of Performance Tests 



Performance tests can, generally, be categorized according to one of 
five types of uses: 



1 . 



To evaluate 
performance . 



proficiency and pr diet 
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successful job 



2. To diagnose deficiencies in an individual’s job performance , 

3. To obtain a quantified estimate of job proficiency as of a 
certain point of time, 

4. To provide a criterion for various uses or applications, 

5. To evaluate competency at various levels of the occupation 
and, thus, serve as a means for selecting the most compe- 
tent individual. 



Oral Examinations for Appraisal of Occupational Competency 

Industry, business and, to some degree, trades and industrial teacher 
education institutions utilize the interview as another means for apprais- 
ing individuals for employment or for admission to teacher education pro- 
grams, Conducted by competent individuals familiar with the techniques of 
interviewing, the interview represents another device for evaluating many 
subjective characteristics. As a measuring device, the interview must be 
conducted as a somewhat informal oral examination. At best, such an exam- 
ination, even when carefully prepared, contains certain weaknesses. Oral 
examinations may provide: "The information which the man has with regard 
to certain elements of his trade. Only insofar as information is the re- 
flection of experience in the trade, obtained at firsthand, will the 
question method give any indication of skill. M 7 

1* Historical Development and Use of Oral Examinations 

The Oral test was extensively used by the United States Army during 
World War I to assign soldiers to military jobs as closely related to their 
civilian occupation as was then possible. To separate the competent car- 
penter, machinist or electrician from those who said they were competent 
in these occupational areas, well-designed questions probably served their 
purpose. 

During the depression years, the Unitec. States Employment Service 
faced the monumental task of classifying huge numbers of applicants accord- 
ing to occupational competency levels. A great deal of effort was devoted 
to the development of oral questions for over 150 trades and occupations 
to aid employment interviewers in determining more logically the competency 
levels of unemployed tradesmen and industrial workers,® In the late 
forties and early fifties, the Employment Services in the various states 
used the questions for job placement and identification of training needs. 
In recent years, the oral examination as the only means of selection has 
been discontinued in nearly all government employment offices. 

Perhaps, this was the result of a widening belief, supported by re- 
cent research results, that there is a low relationship between knowledge 
of an occupation and the actual skill an individual must possess to carry 
out the more complex tasks of the trade. 

However, limited numbers with a contrary view maintain that it is 
possible to separate the comp^et^nt from the less competent individual by 
means of oral questions . ^fljiis^ group assumes that the larger percentage 



11 



of workmen who can perform the work of the occupation at an acceptable 
level of proficiency must be able to talk about their work in trade 
terms, provided the interviewer, or examiner-, and the examinee speak 
the same occupational language. 



2 . The Characteristics of Oral Tests 

In the appraisal of occupational competence one does not need to 
subscribe to either extreme view. However, it must be recognized that 
there are circumstances and conditions when neither written nor perfor- 
mance tests can be administered. Under those circumstances, an inter- 
view may well be the only means available to gain an overall impression 
on an individual’s competence to express himself on matters of his occu- 
pation. Since this is one of the characteristics essential for teaching, 
tha oral examination has value in roughly screening candidates. 

As in all testing, careful effort must be made to: develop valid 

and reliable questions, establish a procedure for recording correct and 
incorrect responses, and to analyze the results when a group of individ- 
uals must be interviewed. Provision must also be made to secure the 
oral questions so that examinees do not exchange oral testing information 
during or after the interview. 

Tn addition to the usual guidelines for preparing good test items, 
the following are emphasized as they relate to oral test items which 
apply to occupational experience and competencies: 

a. The questions must be skillfully formulated and worded 
and must be concerned with familiar processes, materials, 
machines and equipment. 

b. The questions must be clear so that they cannot be 
misunderstood. Suggested elements of correct answers 
should be recorded for use by the examiners. 

c. The terms of the questions should be phrased in the 
vocabulary of the shop or occupation, and must reflect 
the level of difficulty experienced on the job. 

d. The questions should avoid trade terms that are local or 
geographic . 

e. The oral test items should represent all levels of 
occupational difficulty to differentiate among levels 
and distinguish capability. 

f. The questions should aim toward a balance of test items 
representative of each major component part of a trade, 
as determined through an occupational analysis. 

31 
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The United States Employment Service used oral tests almost exclusively 
during World War II to measure occupational competency. This governmental 
service now has standardised sets of oral tests covering over 126 skilled and 
semi-skilled occupations with instructional manuals for their administration 
and scoring. 

Stead and Shartle, who were assisted in this effort to develop oral tests 
to measure occupational competence* reported eight basic kinds of questions: 10 

a. Questions dealing with definitions and calling for short 
descriptive answers . 

b. Questions dealing with, modifications and limitations of 
tools, materials and machines used in the trade. 

c. Questions dealing with the use of tools, materials and 
machines of the trade. 

d. Questions dealing with procedures by which the worker 
goes about doing the operations of the trade. 

e. Questions dealing with the location o f things used , 
processed and assembled in the trade. 

f. Questions dealing with proper names of tools, materials, 
parts and processes. 

g. Questions dealing with the purpose of doing things, using 
things, locating things, and the like, in the trade practice. 

h. Questions dealing with numbers as related to tools, parts 

- of machines, materials and the like. 



3 . Limitations of Oral Tests 

If no other means of evaluating occupational competence is available, 
the oral test serves a useful, limited purpose. The following limitations 
must be considered: 

a. 



b. 



c. 



The answers to oral test items are extremely difficult to 
treat statistically. 

While multiple answers may show that a workman understands 
the whole process, it is rarely possible to give objective 
ratings to the answers. 

There is the possibility that a single key element of a 
process will give an indication of knowledge or ignorance 
of the whole process, 

A 
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4 . Preparation for the Oral Test 



The questions for the oral interview should be developed with the 
following in mind: 

a. Is the test item applicable to the occupation? 

b. Does the answer to the test item reveal or 
involve good occupational practice? 

c. Should the test item be revised, modified or 
supplemented to yield suitable answers, or is it 
better eliminated? 

d. Does the test item differentiate between levels 
of competency and distinguish the individuals 
having the specified competencies? 



5. Oral Test Development Sequence 



Oral test items should be developed according to accepted test devel- 
opment practices with emphasis on the following: 

a* Prepare a test specification in which the major 

occupational areas included in the test are stated* 

b. Test items should be compiler 5 by occupationally 
competent test specialists and be adaptable to 
statistical treatment . 

c. Oral test items should be written, typed or printed 
so that the examiner may refer to them in an unob- 
trusive manner. 

d. The test items should be reviewed for internal 
validity, tested on a sample population, revised, 
etc . 

e. The test items should then be standardized* 

f* Oral test results should be treated statistically 
wherever possible. 

g. The final items for an oral test should be selected 
and the test calibrated. 
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Performance Tests of Aptitude 



In a performance test, the examinee I3 directed to carry out some 
manipulative activity representative of tasks performed in an occupation. 
Like written test3, performance tests can be classified as either aptitude 
or achievement tests according to the purpose for which the test is de- 
signed. The general form of an aptitude test is not much different from 
that of a proficiency test. Its major difference is in the relationship 
between the functions measured by the test and the requirements of the 
job for which performance is predicted. 

Performance tests of aptitude are used before training starts. The 
results are intended to measure the individual's potential skill . The 
test function is to determine the capability of the individual to develop 
through training or the levels of such skill that he can develop, 
and, to a degree, how quickly he may reach a determined occupational goal. 



Performance tests for aptitude consist of some task or group of tasks 
which, although not taken from the job directly for which the test is 
developed, nevertheless, predict success by measuring the aptitude suited 
to the needs of the job. 

Examples of aptitude performance tests are numerous. They have been 
designed to test the individual's ability to assemble small parts, handle 
and place in special location pins with tweezers, fit uniquely shaped 
blocks together— others test the strength of the grip of hands, steadi- 
ness of hand, etc. These tests serve, primarily, to determine an indi- 
vidual s dexterity, his mechanical comprehension, mechanical reasoning, 
and visual acuity among many others. The value of performance tests of 
aptitude is, generally, limited to appraising broad categories of aptitude, 
such as mechanical ability or physicali-motor abilities. 



Such tests have been carefully developed and show the relationship 
between resulting scores and the known aptitudes of the subjects studied 
before the tests have been applied. These tests, since they do not test 
Job knowledge, are suitable in selecting individuals with particular 
aptitudes for training for a particular payroll job. 



Purposes Serve d by Occupational Competency Tests for Future Vocational 
Teachers 



!• Admission to Trade and Industrial/Technical 
Teacher Education Programs 

The teaching profession involves abilities, skills and know-how all 
its own. While there are some individuals who seem to be born with the 
gifts of the good teacher, the larger number must acquire the methods, 

and understanding. Experienced industrial teacher educators have 
observed that the experienced tradesman or skilled technician brings to 
the profession of teaching (Resides his occupational skills and technical 
know-how) an attitude and out*Look on life frequently very different from 
that of the purely col lege- trained teacher. They have, also, noted that 
work habits, relationships among people in an industrialized society, end 
a sense of practical responsibility acquired in the world of work carry 



over into the experiences required by an individual to prepare himself 
fov teaching his occupation. 

While the industrial teacher educator is concerned with the personal 
qualities which are necessary for success in teaching and guides the in- 
dividual through a program of professional studies, he, also, seeks to 
establish evidence of the candidate's level of competence in a trade or 
industrial- technical occupation. Towards this end, the applicant's 
record is evaluated. His prior work experience and recommendations of 
former employers are reviewed; sometimes by a committee of teachers and 
men from industry. 

Although these procedures have worked, much of the information ob- 
tained depends on judgement and general impression. In recent years, 
there has been growing concern about the need for more reliable and valid 
nstruments to establish the level of occupational competence through 
written and performance tests. 

Two points need to be reemphasized at this time; (1) Occupational 
competence in the trades and industries and industrial/technical occupa- 
tions is developed on the job according to a variety of training plans and 
programs; (2) Occupational competency represents th subject content com- 
petency for a course, curriculum or program. 

Well-designed tests, properly validated and with their reliability 
thoroughly established, provide measures that are fair to individual appli- 
cants, provide a means of comparing the demonstrated achievement of appli- 
cants with accepted levels of proficiency, and eventually provide the 
necessary information for the development of practical acceptance norms 
an applicant will have to meet. Tests of this kind provide helpful infor- 
mation which the industrial teacher trainer can apply to correct occupa- 
tional weaknesses in an otherwise qualified candidate, and to accept 

occupationally competent persons into an industrial teacher education 
program. 



2 * Competency Tests for Temporary or 
Permanent State Certification 



Nearly every state recognizes that instructional excellence must be 
rooted in the occupational competence of the teacher. The State Plans of 
at least sixteen states have requirements involving some form of compe- 
tency evaluation of trad , and industrial/technical education teachers 
through testing. Nearly all states have established minimum experience 
requirements as one of the conditions for temporary or permanent certifi- 
cation. State officials concerned with vocational education have recog- 
nized the need for more accurate and effective means for establishing 
occupational competence, and are requiring or encouraging occupational 
competency tests for temporary cr permanent certification. They have 
recognized that reliable test results provide objectiv - evidence of sub- 
ject content capability for certification and will lead to higher 
standards of teaching and learning achievement. Valid occupational com- 
petency testing will, also, improve the professional standing of vocationa 
teachers and the whole teaching profession. 



i 
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3- Occupational Compe t ency Tests for Advanced St anding in 
Collegiate Programs of Study Leading to 
Baccalaureate Degree 



Granting college credits for employment experience is not new. It has 
been a slow and arduous process for colleges to accept the philosophy of a 
flexible system of credit examination. The ideal system of credit examina- 
tion would assess and certify accomplishment on the basis of present perfor- 
mance, irrespective of the route that the individual traveled to achieve 
Such competence. 



Competency in an occupation is considered to include the skills, know- 
ledges and judgements possessed by a typical journeyman. Competency also 
connot-s human traits and capabilities in interpersonal and cwic relation- 
ships... work habits, attitudes, team efforts, etc. 



A national system of occupational competency examinations will permit 
individuals to participate in higher education with advanced standing. This 
recognized advanced standing will remove some of the barriers which now make 
it extremely difficult for vocational teachers to pursue advanced studies 
and achieve their degrees. 



A national testing program will provide the evidence necessary to es- 
tablish standards for the granting of advanced standing and encourage quali- 
fied individuals to strive for further professional improv nent. It will 
enable the institutions to apply the results of their own requirements. 
Ultimately, the national occupational competency testing program cannot help 
but Improve the professional standing of the vocational teacher. To digress, 
it may be stated, parenthetically, that comparable content capability mea- 
sures, developed for all subjects and related areas of the total profession, 
would improve educational and training excellence in general, as well as 
specialized vocational education. 



Importantly, it is here recorded that college credit granted for vali- 
dated competency (validated through practical performance and written theory 
tests) should be equivalent to the credits for college work which leads to 
teaching in the academic areas-the fine arts, the liberal arts, and the 
practical arts. This not withstanding the fact that the greater number of 
practical skills and the related technical knowledge in the trades, industry 
and industrial technical occupations have been developed under actual 
employment conditions in industry. 
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Chapter II 



ADVANTAGES AND DISADVANTAGES OF VARIOUS 'TYPES OF TESTS 
FOR OCCUPATIONAL COMPETENCY EVALUATION 



The test constructor is faced with a number of limitations and obstacles 
in the search for the most objective, valid and reliable instrument. No 
single test can serve the great number of occupations and the great variations 
in skill, judgement and information which represent acceptable journeyman com- 
petence. While there are several broad categories of tests devoted to speci- 
fic competency measurement, each has its strengths and weaknesses. The advan- 
tages and disadvantages of the various types of tests must be considered when 
the test form is selecte 1 for measuring occupational competence. 

I. LIMITING FACTORS IN TEST SELECTION 

Occupational competency requires the measurement of many different char- 
acteristics, factors or qualifications in different occupations. The choice 
of the appropriate test is influenced by what the test is to measure and the 
circumstances and conditions under which the test can be administered. 

The three types of performance tests mentioned earlier, as well as several 
types of writ en u~*rts, each measures some aspects of competence and has its 
place in measurement Which type of test to use fox a given situation depends 
upon a number o x factors such as those identified under the following five 
major groupings: 

A. The Nature of the Trade or Occupation 

1. Does the occupation involve a high degree of manual skill and 
technical informat ion? 

2. Does it require a high degree of technical information and 
relatively simple manipulative skills? 

3. Does it require a high degree of theoretical background and 
technical training? 

B . The Type and Amount o f Machinery and Equipment Required 

1. Is the machinery cr equipment readily available? 

2. Is the equipment, machinery iefnd apparatus required usually 

found in school shops or laboratories, or will industry make 
it available? / 




18 



3. Is the equipment:, machinery and apparatus up to date (like 
nuntericaJ. control equipment, optical measuring equipment, 
electr ica 1 • electronic control devices and measuring instruments 
in a machine shop)? 

4. Are the facilities required readily accessible and available at 
times convenient for taking examinations? 



C . T he Materials, Parts and Supplies N eeded 

1- Is the material required within a reasonable cost range? 

2. Can the material, parts and components be readily obtained? 



D . The Use to be Mad e of the Test Results 

Is the satisfactory passing of the occupational competency examina- 
tion to be used: 

1. To validate occupational experience? 

2. To establish evidence of competence? 

3. For advanced credit for further study? 



E* The Range of Skills^arid Information to be Evaluated 

1. What special aspects of an occupation must be measured? 

2. How are the various levels of competence within one occupation 
to be established? 

3. What are the common elements in a cluster of occupations? 

4. What constitutes testing comprehensiveness for a level of 
competence? 



It must be apparent that the development of proper instruments for the 
evaluation of occupational competence is influenced by many conditions, pract 
cal considerations and judgements. The test constructor must remember that 
the ultimate purpose of the test, or tests, is the evaluation of the present 
competence and proficiency as a tradesman or technician 



II. ADVANTAGES A1TD DISADVANTAGES OF VARIOUS TESTS 



Every measuring instrument contains advantages and some limitations. 
Advantages and disadvantages associated with manipulative (performance and 
written) tests follow: 

A. Advantages of Work-Sample Tests 

1. One major advantage of a performance test is in its apparent 
realism. Performance tests tend to impress upon the candidate a 
more practical measure of his ability and achievement and to make 
him feel that his true proficiency is being evaluated. 

2. Performance tests emphasize operational skills. W.ille skills in 
performing a task can, to a limited degree, be measured indirect- 
ly by means of written tests, work-sample tests provide the 
opportunity for direct measurement. 

3. Work-sample tests are relatively non-verbal, compared to written 
tests. Candidates less facile in writing or reading are not 
penalized as they may be in written tests. Further, the condi- 
tions of testing simulate the conditions surrounding performance 
on the job from simple directions, drawings, etc. 

4. Performance tests can involve a work job as a whole and permit 
the checking of critical steps. 

5. Performance tests are, generally, accepted as a reasonable 
method of relating testing to job duties and responsibilities. 



B . Disadvantages of Work-Sample Tests 

1. Work-sample tests are difficult to develop and construct. This 
is particularly true in highly technical fields; such as, elect- 
ronics equipment operation, maintenance and repair, technical 
design and similar fields. 

2. Performance tests are not easy to administer. In order to assure 
maximum value, work- sample tests require time and competent ob- 
servers, and can be administered to only a few candidates at the 
same time. 

3. Work-sample tests may tie up equipment which may be needed for 
instructional purposes in schools or production when industrial 
facilities are used. 

4. Accurate and consistent scoring methods are difficult to develop 
and maintain. Most scoring on performance tests is the product 
of observation, evaluation and judgement. Inconsistencies and 
errors in judgement, on the part of the examiner, may influence 
test results, unless written directions and measuring scales are 
developed and used by tjhje^examiner s . 



5. Only a limited number of operations or steps can be tested in 
normal time limits. Preliminary research has indicated that 
performance on certain, selected tests may establish overall com- 
petency to a fairly high degree. The validity and reliability 
of such tests depends upon the careful analysis and selection of 
these items which are representative of overall competence. Per- 
sonnel skillful in this area are not readily available. 



Advantages of Written Tests 

!• ^ ne the major advantages of the written test is that it can 
be administered to a large group of candidates conveniently and 
economically. 

2* Written tests permit a wide sampling of facts, information, know- 
ledge of materials, interpretation of specifications, and other 
aspects of the occupation. 

3. Trade content can be checked accurately through the use of 
objective- type questions. 

4. Written tests permit objective statistical treatment of the test 
results. Scoring of test question answers can be achieved by 
clerical personnel working from a preestablished answer sheet. 

5. In the preparation of test questions, it is easier to prepare a 
pool of questions of varying difficulty level for alternate forms 
of the same test. 

6. Written test content can be revised easily. 

7. Computer facilities and procedures make it possible to nrocess 
test results quickly. 

8. Written tests can be readily analyzed and researched as to 
reliability, validity, difficulty level, etc. 



Disadvantages of Written Tests 

1. Occupational skills and procedures involved in the daily work 
of the occupation cannot readily be measured. 

2. Once printed, it is time consuming and costly to make revisions 
in content or in the type of examination questions, 

3. There is major reliance on the measurement of facts and data. 
Practical judgements and the demonstration of skills, handling 
of equipment and tools, etc. do not lend themselves to measure- 
ment by written tests. 



4 . 



Reliance is placed on the individual's ability to read inter- 
pret written material under unusual pressures of time setting- 

5. The planning and preparation of test items requires special know- 
ledge of the occupation to phrase questions in the languors of 
the occupation. Thus, test items are expensive to prepaid. 



III. OCCUPATIONAL COMPETENCE MEASUREMENT 
WITH WRITTEN AND PERFORMANCE TESTS 

A. Value of Multiple-Choice Test Items for Written Tests 

Written tests contain a great variety of different types of 
items: true-false, completion, multiple-choice, matching, short. ax* s Vjer, 

and other problem-solving items. Each type of test item has cert^h ad~ 
vantages and disadvantages. 

A great deal of research carried on in recent years indicate# that 5 
for the written test, mu It ip le- choice test items give the best ro s Ults- 
Well-phrased mu It ip le- choice test items require Reasoning, factual infor- 
mation, computational skills, communication skills, judgement, e£ c . for 
most tests,, this type of question is suggested. Small advantage# ath e r 
forms may have, are outweighed by the overall results obtained £tf { % tests 
consisting of mu It ip le- cho ice test items. The many forms of mul^i^le- 
choice test items and combinations incorporate most of the advant a £es of 
other objective types and provide the necessary flexibility to th e 

form to meet differing occupational requirements. 



B. Need for Both Written and Performance Tests 

For effective performance in any occupation an individual acq u ires 
manipulative skills and certain "know how" or trade information. 

With the rapid development of new machinery, methods and mat^tals* 
many occupations require an increasing amount of technical information* 
While skills are ever changing, they do so mostly towards greater intri- 
cacy and the application of more information. The cutting tools th^ 
machine shop of today are vastly different from the simple carb°t\ and 
high speed materials used in earlier years. The tradesman must b a Ve the 
skills of utilizing diamond tools as well as be able to shape an^ forfl* 
various cutting angles used for different materials. The electronics 
service technician must not only be able to test tubes, transistors and 
solid state circuits, and replace them but, also, he must underst a t\d the 
underlying principles which make them function and trace difficulties to 
their source when trouble develops. 

Evaluating occupational competence requires the development appli- 

cation of both written and performance tests. 
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Chapter m 



TEST DEVELOPMENT PROCEDURES 

I. THE IMPORTANCE OF PROPER PLANNING 

The development of tests for measuring occupational competence requires 
great care, imagination and resourcefulness, as well as sound judgement. It 
is a time-consuming process. The purpose of occupational competency evalua- 
tion is to establish evidence that a tradesman, or industrial technician, has 
the necessary range of skills and the depth of knowledge required as subject 
content to be imparted by the teacher in developing other selected students 
to perform work skills, and obtain technical knowledge essential to employment 
Therefore, the first step, in any test construction project, is to define the 
purpose the test is to serve, set up criteria for the construction of the test 
plan the procedures to be followed, and determine the specialists who must be 
involved. Careful preplanning is essential. 



II. OUTLINE OF STEPS IN PLANNING WRITTEN TESTS 
A. Sources of Related Job Classifications 

Increased specialization in many occupations makes it difficult to 
recruit the all-round tradesman who entered his occupation via the appren- 
ticeship training program. Special vocational education school programs 
for youth and adults, inplant training, the pick-up method, and other 
patterns of training are avenues chrough which people become proficient 
in their occupations. To enable people with widely varying backgrounds 
of preparation to become teachers, a test must sample the skills and know- 
ledge of a cluster of related job classifications. Those clusters can be 
identified from such sources as: 

1. The Dictionary of Occupational Titles or other Standard 
Industrial Classifications , 

2. Joj descriptions and classifications from employers and 
occupational studies of labor, management, etc., and 

3. Uni«^ States Office of Education taxonomy of vocational 
education fields and major occupational groupings in job 
clusters . 



. Proced ures for Determining Occupational Competencies to be Me asured 

1. Establish the purposes and rationale behind the development 
of the occupational competency examination. 

2. Chart the overall project, as suggested in Table I. 

3. Select qualified personnel to serve as a test advisory and 
development canrad.ttee. 



4. Identify jobs by title and specifications that are typical of 
the occupational area and give the range o: competencies re- 
quired for the whole occupational area„ For instance, Table II 
identifies selected job titles for the electronics industries 
occupations (communications) that range from single skill oper- 
atives to highly skilled professional persons. 

5. Group the job titles, which require similar skills, technical 
knowledge, and other occupational responsibilities and duties, 
into job clusters on different levels (Table III) . 

6. Determine the levels (according to objectives to be achieved 
by the test) for which the occupational competency test is 
to be developed. 

7. Make job analyses of jobs in the cluster in term of: 

a. Manipulative skills 

b. Trade theory and judgements 

c. Trade communication (technical language specifications, 
drawings, etc.) 

d. Computational skills (mathematical calculations and 
reference tables) 

e. Applied principles of sciences 

f. Technological knowledge needed 

g. Industrial safety, health and hygiene practices 

h. Other information essential to the occupation 
C . Job Description and Test Specifications 




1. Prepare job descriptions. These should define job skills and 
accompanying trade theory for the performance part of the test. 
Related trade/ technical information, knowledge, judgements, and 
other occupational competencies the potential teacher must know 
should be described as a guide for the written part. 

Table IV describes special occupational items for which an 
occupational competency examination is to be prepared for the 
electronic industries occupations (communications). 

2. Establish the scope of the test in terms of level and range 
of skills and information to be evaluated. 



Determine the nature and format of test items that should be 
prepared for the written part of the test. 
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3. 



4 . 



Set up a specification table to roughly establish the compo- 
nent parts of the occupational competency test for each main 
and subdivision. The number of proposed test items to con- 
sider should be noted. 

> 

The structure of Table V suggests columnar groupings for 
which test items should be prepared. 



Alternatives for Preparing a Job Description to Establish Test Content 

At times, it may not be possible to carry out a job cluster analysis 
and complete the required amount of detailed search and analysis. Even 
under restrictive conditions, a job description should be prepared. While 
it may not be as comprehensive, the description will help to identify test 
items that are not easily determined by other means. 

The following techniques may be used to prepare a job description: 

1. Actual on-the-job on-site observations may be made of the 
work to be done by a trained person in the occupation. Such 
observations are made by occupationally competent persons, 
together with a test specialist. 

2. A representative group of experienced craftsmen within the 
occupation may prepare a job description. 

3. The job description of the job cluster may be prepared as 
described earlier. 



In any case, the job description is another tool which provides di- 
rection and assistance * o the test developer. 



Test Items, Test Construction and Evaluation 

1* Establish procedures for electronic data processing relating 
to test construction, administration, scoring, recording, 
analysis and evaluation. 

2. Assess the test structure and the number of test items 
needed to achieve the test objective. 

3. Prepare a pool of test items with answers. 

4. Evaluate each test item and revise as needed. 

5. Arrange test items in proper sequence within the occupa- 
tional competency test. 

6. Prepare a Handbook for examiners with answer sheets. 
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Table II. 



SELECTED JOB TITLES FOR ELECTRONIC INDUSTRIES 
OCCUPATIONS - COMMUNICATIONS* 



Job T j.t les 



D.O.T. Code 



Electronics Assembler 
Electronics Assembler Development 
Electronics Mechanic 
Electronics Technician 
Radio Mechanic (and ind . ) 

Radio Mechanic - Troubleshooter 
Radio Mechanic - Repairman 
Television - Installation 
Television - Service & Repairman 
Television - Chassis Inspector 
Television - Transmitter Assembler 
Electronics Sensing Equipment Assembler 
Component Inspection Technician 
Customer Engineering Specialist 
Radioactivity Instrument Technician 
Electronic Mechanic Apprentice 
Electronic Sound Technician 
Public Address Serviceman 
Electronic Technician 

Systems - Testing - Laboratory Technician 
Radio Engineer ) 

Television Engineer) 

Radio Equipment Installer 
Radio Repairman Domestic 
Radio Mechanic Helper 
Electrical and Radio Work-up Man 
Television Repairman Apprentice 



726.781 

726.781 
628.281 
003.181 
823. 281 
720. 281 

720. 281 

823.781 
720. 281 
720.687 

726.884 

716.884 

828.281 
828. 281 
828. 281 
828.281 
024. 288 
823.281 
003.181 
003. 181 

003.081 

823.281 

720. 281 

823.884 
325.381 
720. 281 



T.V. Audio Man ) 

T.V. Transmitter Maintenance ) F.C.C. 1st class license required 

Microwave Technician ) 

Mobile Communications Equipment ) F.C.C. 2nd class license * equired 
Two-Way Radio Communications ) 



* Derived from the Dictionary of Occupational Titles 
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Table III. JOB CLUSTERS FOR FIVE MAJOR LEVELS C'.’ OCCUPATIONAL PROFICIENCY FOR 

ELECTRONIC INDUSTRIES OCCUPATIONS (COMMUNICATIONS) 
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Table IV. JOB DESCRIPTION (OCCl TP A T 7'>NAL COMPETENCIES) FOR 
ELECTRONICS OCCUl-^i: COMMUNICATIONS 



General Comments 



For an electronics industry occupational competency testing program to be 
used nationwide, it is essential to include test items of common use in var- 
ious sections of the country rather than specialized practices of a particu- 
lar area. The competent tradesman should be able to perform his work regard- 
less of geographic location or national origin. The fallowing job descrip- 
tion should be used as a guide for test development: 



Job Duties; Skills and Related Trade Theory 

• Repairs, adjusts and maintains electronic equipment such as: radio receiv- 
ers (both AM and FM) , audio amplifiers, tape recorder units; communication 
receivers, transmitters, receivers; T. V. receivers. 

• Must locate faults, replace defective components, test, align and cali- 
brate equipment for; power output, correct frequency, distortion levels, 
audio and video quality; measure current, voltage, resistance wattage and 
frequency; test all components, test tubes, diodes and transistors; trace 
ci '®“y ts t nd make 311 nece ssary corrections. Installs, tests, adjusts and 
modifies inter-communications systems, using the instruments of the trade. 



Technological Informa tion and Knowledge and Trade Judgment 

• Must know electrical fundamentals of AC/DC capacitance, resistance and 
inductance, series circuits, parallel circuits, series-parallel circuits, 
resonance, magnetic theory, transformers. 

• Must know electronic fundamentals of solid state diodes, transistors, 
vacuum Lubes; diode, transistor and vacuum tube characteristics curves 
power supply circuits, A.F. amplifier circuits, RF amplifier circuits, 
video amplifier circuits, microwave circuit techniques, oscillator circuits, 
amplitude modulation, frequency modulation, pulse modulation, negative 
feedback circuits. 

• Must know troubleshooting methods; signal injection; signal tracing, 
voltage analysis and resistance analysis techniques. 

• Must know basic power distribution system; 117- volt single phase, 220- volt 
single phase; two and three phase systems. 

• Trade judgment. Must exercise judgment in the selection of the most suit- 
able procedures for the location of faults, in the selection of the appro- 
priate test equipment, in the repair techniques and the replacement of 
substitution of parts of modular units when needed. 
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Table V. TEST ITEM PLANNING FOR INDUSTRIAL ELECTRONICS- COMMUNICATION 

OCCUPATIONAL COMPETENCY EXAMINATION 
(Suggested Columnar Headings Under Which Skills and Related 
Technical Knowledge are Identified. - Not to Scale) 
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■* . Review the test in committee for structure, time, relevance, 
etc. , utilizing the services of a test and measurement 
specialist. 

8. Determine a representative sample group for pilot testing 
the examination. 

9. Field test, evaluate and refine. 

While net complete and detailed, the basic steps outlined in this 
Section II must be followed. Only after the test items have been field 
tested can final revisions be made. Weak, inadequate or duplicate test 
items can be eliminated. Thereafter, an instrument can be developed 
that gives reasonable assurance of being valid and reliable. 



III. PLANNING PERFORMANCE TESTS 



General Consider ations 

The steps involved in planning the performance test are somewhat 
similar to those that were outlined earlier. However, since work sample 
performance tests are usually concerned with some overall job situation, 
its scope, content and requirements warrant separate consideration. 

Some individuals believe that any performance in which tools are 
used, a machine is operated, parts are checked and replaced, or things 
are manipulated, represents a satisfactory performance test. Such a 
test may seem to have a great deal of face validity, since it appears 
to duplicate the job f Atuati;. n so well that there seems little doubt 
that it measures the intended purpose. 

This may not reaily be so. Unless -reat care is taken and use is 
made Ol the job cluster analysis for those skills and methods which are 
vital to the job performance, and provision is made for step-by-step 
procedural planning, standardization of administration, and objectivity 
•n scoring, the test can easily become an unreliable instrument. As 
such, it reflects merely the judgment of the administrator and it may 
not be val 

Steps in Planning a Performance Test 

The following steps will lead to the design of an effective test 
regardless of whether the performance test involves an actual work job, 
a simulation situation, a combination of these two methods, or recog- 
nition procedures: 

1. A work sample must be made up of representative ele:. ats 
of the job titles in the job cluster. 

2. Eacl element in the work sample uuld per_ t :i on 

in performance and be applicable to ot h -k ^ods. 



3. A variety of work samples should be tried. Statistical 
analysis should be used to determine which element ^ discri- 
minate, and that no important phase of the job has been 
eliminated in the process of condensing the test. 

4. The tasks, selected for the work sample must be representa- 
tive of the job cluster and job level and require suffi- 
cient time f.o lead to reliable scores. 

5. The tasks chosen should command respect and establish the 
confidence of the individual engaged in the job as being 
representative of the occupation. 

6. The materials, tools and equipment should be reduced to the 
smallest practical quantity and should be capable of stan- 
dardization so that the same test may be given under uni- 
form conditions. 

7. The performance should involve as little repetition of 
identical procedures as possible. 

8. A preliminary tryout of performance tests should always be 
made using experts as students to detect possible problems 
or difficulties. 



IV. EVALUATING THE PERFORMANCE TEST 



A. Work Method or Finished Product 



In evaluating a perforrance test, the question often arises as to 
which is more important; the finished product or the work method used and 
the work habits applied to the completion of a job. For most occupations, 
the either/or approach is not applicable. Each occupation has its own 
characteristics . There are great variations in skill and knowledge among 
the many occupations for which potential teachers must be recruited. 
Careful analysis is required to which approach is more important and 
should receive greater weigh 1- . 



For example, if the occupation involves troubleshooting, of electri 
cal circuits , it is posssible to accidentally stumble upon the trouble 
and then correct the defect without the systematic process of testing 
and elimination. The equipment may perform according to specifications 
without the examinee really having do.ionstraf od that he ossosses the 
skills of analysis and step-by-step checking. On the other hand, in the 
production occupations (machine trades, sheet metal trades, etc.) the 
nature of the work and the finished product make "accidental aids” virtu- 
a ^y impossible. It is easier to evaluate the finished product object- 
ively than to appx*aise work method and work habits. 
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Identifying Craftsmen Characteristics 



Although occupational competence tests are not intended to evaluate 
and predict teaching potential, it is important for the tradesman and 
future teacher to demonstrate those work habits and procedures which dis- 
tinguish the good journeyman from a less competent individual. Eventual- 
ly* these are the very things he will have to teach and they, thus, repre- 
sent an important area of competence for his future career. 

Therefore, the work sample selected for the performance test must 
permit an evaluation of work habits as well as permit an objective eval- 
uation of the finished product. The choice of the work sample is further 
influenced by the length of time reqi ired for its completion, the amount 
of equipment required and accessible, availability of testing personnel 
and testing facilities, funds, etc. 

Equally important is the choice between a single long job which in- 
volves selected skills and methods of procedure or a number of smaller 
assignments with each requiring special, vital skills. Again, the nature 
of the occupation will largely determine this choice. It will be easier 
to design a work sample which does not need to be fully completed for the 
machine trade than for a service occupation. For example, the skill of 
using layout procedures, operating a drill press, a grinder and a miller 
can well be combined in one work sample. For another, it is possible, 
to combine lathe work, heat treating and grinding. A single work sample 
arrangement would require more bulky equipment of various kinds and be 
thus more limited. 

In any case, both method and product must be evaluated. The nature 
of the occupation determines where the greater emphasis should be. 



V. PARTS OF THE OCCUPATION REQUIRING EVALUATION 
Aspects that Represent Typical Jobs 

The parts of the job to be measured in a performance test are those 
which represent the occupation as a whole. They should be selected on 
the basis of how critical they are to performance in actual wage-earning 
employment for the proper operation of machines, apparatus, equipment, 
etc. Many tasks in a job may be of a routine nature and a great deal of 
time may oe devoted to them in the shop and laboratory. 

In the interest of time and economy of test administration, it is 
important to reduce or eliminate repetition or duplication of certain op- 
erations and concentrate or those more critical on the job. For example, 
a ^ ter a lathe job has been completed and the examinee is required to dem- 
onstrate his skill on the milling machine by cutting a gear, it w.vilu be 
a waste of time and useless repetition to have him complete the whole 
gear. Cutt i.g th-ee or four teeth would demonstrate whether he knows 
how to do it and will enable the examiner to measure the quality of the 
work . 



Si' -arly, in a test of radar ^nkrintenance, including the calibration 
of range marks f O’- search equipment is unnecessary because search type of 
radar can be operated under combat conditions without close- range calibra- 
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tion. The test might, also, omit determination of band width by the use 
of sweep generators because band width of most radar sets is usually fixed 
and cannot be changed without changing components. 

On the other hand, power measurements must be included because the 
maximum range of the radar set is partially determined by the amount cf 
power transmitted. Unless a repairman can determine the amount of power 
transmitted, he cannot know whether the weak or few video signals receive 
are due to a defect in the transmitter. In like manner, the test should 
include the use of a signal generator because the radar repairman needs to 
know if the radar set is able to pick up and display weak signals from the 
maximum range of the set. Both of these tasks determine n* performance 
figure of the radar set and are crucial to its return to die proper opera- 
tion in the shortest possible time. 12 



B. Establishi n g Critical Items to be Measured 

The items to be measured must be derived from the occupational anal- 
ysis. This should facilitate the determination of the operations to be 
included in the work sample. The analysis chart would show the repetition 
of various operations in the different job clusters. It becomes then a 
matter of judgement to distinguish the critical and important items from 
those that are common to the various job classifications and less impor- 
tant for inclusion in the test. 



VI. TERMS DESCRIBING AREAS FOR SPECIFIC TEST CONTENT 



A . Variations in Test Terms 



Repeated reference has been made to job analysis, and several ap- 
proaches have been described. Some terms, found in the literature, are 
confusing. One author, for instance, calls the breakdown of the steps in 
thre-u cutting an analysis of a job; others refer to the same item as task 
analysis. Still others use such terms as cognitive and perceptual objec- 
tives for making an analysis. 13 

Under the cognitive approach are such subdivisions as knowledge, 
understanding, application of knowledge, application of understanding. 

The perceptual approach covers such areas as sensation, figure perception, 
symbol perception, and perception of performance. 14 ^ further distinc- 

tion is made by Simpson, who discusses the psychomotor domafn. 13 
approaches the problem in terms of a proposed taxonomy of the perceptual 
domain. 1® Krathwohl discusses the affective domain. 1? While these ap- 
proaches arc different in terms and essential toward developing new ap- 
proaches to teaching and the evaluation of learning, they are, at this 
stage;, pi dominantly academic approaches. lor aiding the test instructor, 
faced with the immediate task of developing of occupational competency 
examinations simpler definitions are needed. 
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B. Terms Defined 



The following simplified definitions of terms are used in this Hand- 
book. Examples of each of the four major terms are given to identify ap- 
plications in a number of different occupational settings. Content relat- 
ing to each term is derived from the job analysis. 

^■ # Operat ion - a basic skill involving the correct manipulation of 
tools, machines, equipment, apparatus and supplies to complete a 
specific task according to procedures and practices followed in 
a trade or industrial/technical occupation. 

a. Laying out object lines to describe a part or mechanism 

b. Straight turning a work piece oil a wood lathe 

c. Testing a series circuit for vt ge 
D. Assembling rings on a piston 

2* Rela ted Technical Information - information necessary to carry on 
the work of the trade, involving basic principles of mathematics, 
science, drawing, etc., applied to practical trade practices. 

a. Electronic theory, as applied to transistors and solid state 
circuits 

b. P^nciple of combustion, as applied to power systems 

c. Mechanical advantage, as applied to pneumatic and hydraulic 
clamping devices 

d. Theory of atmosphere control in heat treatment of metals 

Trade Theory - involves technical content knowledge, directly 
related to an operation. 

u. Letter and number drill size systems 

b. Grinding wheel specification's and surface finishes 

c. Transistor functions, capacities, structure 

d. Determining parts from m/gn. catalogues for a certain year 
car model 

^ m Gener al Trade Knot, .Ledge - this term efers to ancillary content, 
work itt itudes nnd personal r elat ionsnips . 

a. Machine and tool safety 

b. Content bearing o- health and industrial hygiene 

c. Employer, employee, and community relations 

54 




35 



VII. SUMMARY EXAMPLE: CONSIDERATIONS FOR THE DEVELOPMENT OF 

WRITTEN AND PERFORMANCE TESTS FOR ELECTRONICS INDUSTRIES 
OCCUPATIONS- COMMUNICATIONS 

Purposes : To provide measures of the technological knowledge and 

ability and the trade skills through which to establish 
the occupational competency of an. individual. The tes* - 
results may be used alone or in combination to; 

1. admit examinees to "ndustvial teacher education 
programs , 

2. to determine the extent to which college credit 
should be granted when pursuing an undergraduate 
or graduate degree program, 

3. diagnose occupational strengths and weaknesses, and 

4. certify capable e- a ftsmen with professional prepar- 
ation as vocational teachers. 

Test Standards : General - 

The tests must represent the best possible instruments 
acceptable for validity, iability , comprehensiveness, 
currency and economy of administration. The final in- 
strument should evaluate occupational competence for 
several levels of job clusters-- each on the achievement 
level requirements of industry. 

The job clusters, including che semi-professional tech- 
nician level, represent the range and levels for which 
occupational competency examinations should measure the 
potential teacher's competency. 

Methodology: General - 

The final tests should be based on an occupational ana- 
ysis. Job titles in occupational constellations in 
the electronics industries-communicat ion and related 
occupations should be clustered according to various 
levels of competency. The levels are established from 
operative to semi-skilled craftsmen, skilled craftsmen, 
technician in industry service, to semi-professional 
and professional engineers. 

Approach to Occupational Competency Test Deve lo pment: 

1. The job clusters at each level require an analysis 
of common elements. These elements involve manipu- 
lative skills, technology, mathematics, principles 
of science, communication (drawing, reading of 
spi cifications, etc.), design, and materials in- 
volved in tne job performance of the occupation. 



2 . 



4 . 



E. Test Utilization- 
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Job Description : 



Electronics Industries Occupation - Communications: 



1. General Comment s: The electronics industries in- 

clude a number of jobs differing in detail and dif- 
ficulty level which, nevertheless, contain substan- 
tial common elements.. Some are narrowly special^ 
ized; others require a broad background of informa- 
tion and skill. The prospective teacher must have 
the occupational competence to teach p reemployment 
classes involving fundamental skills and concepts; 
advanced modern trade knowledge, skills and theory; 
preetnp laymen t classes for special industrial em- 
ployment; occupational extension classes to extend 
the work skills and technical knowledge of the 
skilled worker; apprentice classes; and foremen and 
supervisor classes . 

The Occupational Competency Examination must measure 
the individual’s occupational competency required 
to teach at all levels within secondary schools, 
post-secondary institutions, and adult education 
programs . 

2* Earlier comments and illustrations (refer to the 
preceding Tables I through IV) related to job 
cluster practi 's on multiple levels of employment 
and other job cha 'cteristics and descriptions. 

The job description, naturally, includes job duties, 
skills and related trade theory, technological in- 
formation and knowledge, and trade judgement. 



Occupational Competency Measuring Instruments : 




Written Test ; Knowledge of trade Information, pro- 
perties of materials, principles of design, and 
operating principles of equipment; computational 
skills; and the abilities to interpret blueprints, 
wiring diagrams, schematics, and specifications, to 
apply scientific principles, and to exercise trade 
judgement can be evaluated through written tests. 



It is important that the questions be objective in 
nature with correct answers provided. The number 
of test items should be sufficient for each job 
cluster to adequately measure the individual’s 
competence . 



2- .^x ance Test : The varying nature of service, 

nibxifenance and repair make the choice of jobs for 
evaluating a craftsman’s competence difficult. 
Every job to be done should invo7.ve essential key 
skills without which a job could not be completed. 
In the electronics field the performance test may 
involve a: selection of short jobs on different 
kinds of apparatus, or a single job with a number 
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of difficulties involving the use of meters, equip- 
ment, tools, etc. Thought should be given to prac- 
ticality, facilities, and the availability of units 
on which work can be done. At all times, the jobs 
selected for performance and evaluation should in- 
volve a range of practical skills, which, when pro- 
perly performed, establish the occupational compe- 
tence of the prospective teacher. 



i* Scoring Methods: 1. Written Test : The scoring of written tests should 

be accomplished by clerical persons. A special key 
form that permits the identification of right or 
wrong answers will be provided. It is preferred to 
develop a test that can be scored by electronic 
data processi\ig equipment. Scoring must be done 
efficiently, accurately and confidentially. 

2. Performance Test : Evaluation of the tradesman 1 s 

occupational competence involves consideration of: 



a. The skill with which he handles the instruments, 
apparatus, and hand tools of the technical 
occupation . 

b. The evaluation should take account of the work- 
ing methods and procedures of a candidate and 
the performance of the completed unit. 

c. The skill he demonstrates in using reference 
materials, technical manuals, specifications 
and directions. 

3. The evaluation should be made on troubleshooting, 
checking and tracing circuits, tracing various 
stages of a unit, identifying the trouble, replac- 
ing a defective part or unit, aligning, calibrating 
or testing a unit for proper performance. 

4 . Items for Skill Evaluation of the Examinees : 

a. Preparation and set-up of units to be serviced 

b. Arrangement of instruments, meters, tools to 
be used 

c. Orderly and logical procedures from unit to 
unit 

d. Steady work - step after st ep . . . . occupat iona 1 
confidence 

e. Personal and machine tool' safety 
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f. Neatness in working 

Recording of all necessary information 

h. Checking measurements carefully and accurately 

i* Demonstrating capacity to use reference 
materials properly 

j. Checking proper performance of the test job 

J • Ra t i ng F or m : rating form should be designed which permits unobstru* 

sive recording of specific observations; contains numer- 
ical or letter values for the items to be observed, and 
permits easy and quick computation of test results. 
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Chapter IV 



CONSTRUCTION OF TESTS 
I. PERSONS INVOLVED IN TEST CONSTRUCTION 
A . The Role of the Subject Specialis t 

Unfortunately., there is no universal agreement as to what constitutes 
vocational success, which is the goal of prediction in the case of Civil 
Service tests. What is accepted as evidence of success is largely a mat- 
ter of the combined judgement of a number of competent persons as to what 
variables to include in establishing adequate measures. 

The test developer knows the principles of personnel measurement; ap- 
propriate methods and techniques for determining whether a test serves the 
purpose for which it is designed, and whether test results provide accur- 
ate and reliable evidence of occupational competence. His knowledge and 
understanding of the occupation may be of the "talking knowledge" kind. 

It is necessary, therefore, to engage the assistance of an individual 
who "not only knows" his trade but has mastery of the essential skills. 

In short, he must be occupationally competent. He must know which skills 
and knowledge are vital to the performance and the main problems encounter- 
ed in his occupation and at what level. 

The subject specialist provides the occupational content for the test. 
He makes the job analysis, plays an important role in the content of the 
test items, and assists in establishing what constitutes a measure of 
occupational competency. 



B • The Role of the Vocational Teacher 



Recruiting tradesmen experienced in test development is not an easy 
task. Availability, time and cost are often limiting factors. Vocational 
teachers who are presumed to be masters of their occupati:.* with an under- 
standing cest practices can provide valuable assistance. Under proper 
guidance and direction, they can make a valuable contribution to the test 
content . 

However, a word of caution is in order. Since teachers develop tests 
for use with their own students, they tend to emphasize instructional con- 
tent rather than prevailing occupational practice. Concerned with teach- 
ing methods and procedures, they frequently recommend content which they 
believe a tradesman should have rather than what may actually bo required 
on the job. Often- they include subject content they, themselves, have ac- 
quired after they started teaching. Furthermore, there is a tendency 
towards developing test items chat may be somewhat n schoo lish n in language 
and application. 





The Role of a Committee of Specialists 



C. 



1. Function of the Committee 



The process of developing and constructing occupational competen- 
cy tests is a cooperative undertaking. Representatives from the occu- 
pation or subject specialists need to be brought together. In select- 
ing such individuals, it is important to have representation from var- 
ious branches of industry in which the trade is practiced. This re- 
duces the possibility of the resulting examination being biased in one 
direction. 



For example, in electronics communication, it would be unwise to 
have only representation from radio and television service maintenance. 
Similarly, in the machine trade, it would be inviting bias if only 
tool and diemakers were represented and no machinists included. It is 
of equal importance to have representatives from several different 
manufacturers rather than one large organization. Such a committee 
should include teachers of the occupation for which a test is being 
developed. 

The committee’s major function is to determine the subject con- 
tent, areas tJ be covered and. the development of each item. The test 
specialist serves to recommend the test form and the technical test 
merits of each item. Sin. e occupational competency deals with special- 
ized content and skills, each test item should be prepared by the sub- 
ject specialist. This first draft should then be appraised for tech- 
nical defects by the test specialist. 

An item which covers an important skill, fact, or judgement may 
still be unsuited for differentiating between a competent, or less 
competent individual if it is phrased ambiguously or it is too famil- 
unfamiliar to the group being tested. Close cooperation be- 
tween subject and test specialists is important in every step of test 
development. 

2 . Size of Committee 




The size of the committee is critical. If it is too small, there 
will be inadequate representation of all facets of the occupation. 

Too large a group may slow down the process, increase developmental 
costs and be cumbersome. While it is possible for one subject special- 
ist and one test specialist to construct a test, it is more advantag- 
eous to pursue the following in organizing a committee: 



a. The subject; specialists should represent several areas 
of the occupat ion . 

b. Vocational teachers should be represented. 

c. The composition of the committee depends upon the nature 
of the occupation and difficulty, or complexity, of its 
content. There should be a balance among the representa- 
tives. 



Working groups tend to Uefcbme 
ive as the size, hicreases. 
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less efficient and products 



d. 



e - While it is not possible to give definite numbers, exper- 
ience has shown that groups of 3 to 6 function w_ll in 
developing an occupational competency test for a major 
occupational area. 

f. Should a larger group be needed, their output becomes more 
effective if the group is subdivided into emaller working 
groups. 

3 * Commi ttee , Direction and Working Procedures 

A committee makes valuable contributions toward test development, 
as the broader experiences represented should assure wider coverage 
of a given field. However, there is always the hazard that in a large 
group much effort is spent in fruitless discussion. It is important 
that some preliminary work be done by the test specialist. 

The great variety of occupations make it nearly impossible to 
establish a rigid procedure. Care must be taken that any preliminary 
plans do not limit the efforts of a committee. At the same time, 
certain directions assure more effective products ity. 

The actual work expected of the committee, which may include the 
following, shou d be established beforehand to assure greater product- 
ivity of the committee’s efforts. 

a. The job analysis must be reviewed* 

b. Test specifications are to be reviewed, revised or 
actually established. 

c * A t t grid, or pattern, or outline must be established as 
a 1 for deriving the number of test items needed for 
eat ubdivision of the occupational job cluster. 

^dures for developing test items must be established. 

e. Ev uation must be made of factors established for judging 
te C results; written and performance. 

Jl. Suggested contacts in industry for testing the examina- 
tions are to be identified. 

g. Recommended standards to be employed for judging test 
achievement are to be developed. 



II. FUNCTION OF TEST SPECIFICATION 
A. Range of Test Specifications 




A test specification may be a comprehensive covering of test purpose, 
standards, methodology and test utilization. It may include the scope, 
job description, job cluster, types of tests desired, scoring methods and 
items recommended for evaluation. It may also include a topical outline 
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recommended for evaluation. It may also include a topical outline list- 
ing the trade content for written and performance evaluation and an esti 
the number of test items recommended for evaluation. 



Table VI . SPECIFICATION FOR WRITTEN TEST - ELECTRICITY (Partial) 



Main Divisions 


Factual 

Knowledge 


Application 

of 

Knowledge 


Understand- 
ing of 
Processes; 


Ca lcula- 
t ions 


| Technical 
Interpre- 
tations 


Total 

Test 

Items 


D.C. Circuits 














Series 

Characteris- 
t ics 


2 


3 


2 


3 


8 


18 


Parallel 


2 


3 


2 


4 


8 


19 


Comb inat ioits 


2 


6 


5 


8 


8 


29 



Other forms of specifications may substitute processes, o: operations, or 
skills. Table VI merely indicates a pattern from which can be derived the es- 
timated number and application of test items that are to be developed. 

Many other variations of test specifications are possible. In academic 
tests, such a specification is often called a Test Plan in which objectives 
and different subject areas are listed (Table VII). 

19 

Table VII. TEST PLAN LISTING OBJECTIVES AND SUBJECT AREAS 



Ob j ect ives 


Subject Areas j 


Foods 


Digest ion 


Assimilation 


Knowledge of 
Terminology 








Knowledge of 
Facts 








Knowledge of 
Trends and 
Sequences 


1 


;3 
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Where it is desirable to establish different levels of occupational com- 
petence, a test specification (also called a.n outline) may be helpful 

(Tab c VIII) . The grid in a multiple level outline shows some common and some 
unique areas that must be tested. 

Table VIII. MULTIPLE- LEVEL OUTLINE 



Areas of Knowledge 
and skills 


Novice 


Apprentice 


Journeyman 


Master 


I 


25 


16 


10 


_ _ 


IX 


10 


10 


10 


10 


III 


10 


\ 

15 


15 


15 


IV 


20 


15 


10 


10 


V 


20 


20 


20 


20 


X 




10 


15 


15 


XI 






10 


15 


XII 






5 


20 


XIII 








10 


XIV 








20 


Total Test Items 


85 


86 


95 


135 



Whichever form is employed for planning an examination, it must be appar- 
ent by now that the comprehensive specification provides the overall pattern. 
Eventually, it will be necessary to develop a more specific blueprint to serve 
as a reference and pattern for the number of test items required for each cat- 
A test developed in accordance with definite specifications is more 



egory . 



likely to yield a score that is meaningful than that of a test whose items are 
assembled at random. 
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III. FORMULATION OF TEST ITEMS 
Choice of Test Items 

Are essay questions more reliable than obj ect i ve- type test items? 

Are multiple-choice items better than true-false items? The relative 
merits of different types of test questions, or items, were long a popular 
debate in educational circles. We now know that there is no general an- 
swer to such questions. There is no one type that is inherently superior 
to other types. How well an item 145, Constructed is more important than 
the type the item represents. Each type of item, moreover, has certain 
strengths and weaknesses. The choice of item types for classroom examina- 
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tions, therefore, evolves from the special advantages and disadvantages 
for each of the purposes the teacher has in mind." 



B. Popularity of Multiple-Choice Test Items 

For purposes of test construction for occupational competence, fac- 
tors of economy of administration, effectiveness of scoring and statisti- 
treatment °f data are important considerations. Among the many objec- 
tive-type tests, the multiple-choice has gained more and more popularity. 
The multiple-choice form is equally as good as the true- false or short an- 
swer item for measuring factual knowledge. Furthermore, it has been found 
to be excellent for measuring understanding and for the ability to apply 
concepts and knowledge to unique situations. 



The military and civilian services are largely committed to the use 
of multiple-choice items, since they consider multiple-choice items super- 
ior to other types. Since the multiple- choice type is adaptable to the 
best answer approach, it avoids the ambiguity associated with the applica- 
tion of a standard of absolute truth, which is the chief weakness of true- 
false test items. Also, mult iple- choice type items are independent of 
each other, sj the problem of finding a number of parallel relationships, 
which frequently cause difficulty in the matching type of items, is avoided. 



Other advantages follow. If adequate checks of accuracy of statement 
are made before the items are used, the multiple-choice form will usually 
avoid unforeseen difficulties due to the subject's misunderstanding of the 
intent of the item. 



Multiple-choice items permit the use of standard machine scorable an- 
swer sheets. Since most testing programs are equipped to deal with multi- 
ple-choice answer sheets, standardization in a number of different estab 
lishments can be undertaken as routine practice. 

C. Changing Alternative Types of Test Items 



Newer test item 
quire a best answer 
with each item. The 



types include changing-alternative items. These re- 
to be selected from several alternatives which change 
most usual form is the multiple- -hoice item. 



The changing-alternative type of item is adaptable to testing higher 
mental processes, such as inferential reasoning, fine discrimination, and 
knowledge of isolated facts. It is the most flexible of its kind of test 

item available for varying types of mental processes on the basis of sub- 
ject matter. 



D . Summary of Ob j ect ive- Typ e Tests 

For all its advantages, there are, nevertheless, occasions when other 
ypes of objective test items may serve a definite purpose. Table IX sum- 
manzes the more commonly utilized objective test item forms, their advan- 
tages and disadvantages, and where they may be used in occupational compe- 
tency testing. (Table lx, pages 47-49) 
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Table IX. SUMMARY OF COMMON OBJECTIVE- TYPE TEST ITEM FORMS AND CHARACTERISTICS 
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• Many items cannot be e Recognition or ider.ti- 

conveniently grouped fication of parts and 

relationships 




Verbal facility and a Tends to be confined 

memorization of facts to identifying, naming 

and associating facts. 
Relatively unfit to 
test understanding 



IV. VARIATIONS AND USE OF MULTIPLE- CHOICE TEST ITEMS 



For this Handbook, the multiple-choice test item is 
most suitable form. Thus, this portion will be devoted 
various ways in which multiple-choice items can be used, 
formulated and the various difficulty levels established 



recognised as the 
to a discussion of 
how they are best 



the 



The multiple-choice item consists 
incomplete statement called the stem, 
the one best, or correct, answer must 
proaches possible can be seen from the 



of: (1) an introductory question or 

and (2) two or more responses from which 
be selected. Many variations in ap- 
fo llowing examples and descriptions: 



A* The Stem as a Question or Incomplete Statement 



The stem of multiple-choice items may be classified according to the 
form of a question or an incomplete statement. This is illustrated in 
the following two examples: 



1 . Quest ion- form example : 

A square has how many sides? (Stem) 

a. Five c . Three ) „ 

b. Six d. Four ) Res Ponses 

2 . Incomplete- statement example : 

The number of sides contained in a square is (Stem) 

a. Five c . Three ) 

b. Six d. Four ) Responses 



Responses Requiri Selection of One Right Answer or the Rest Answer 

u MU ,lu iple ' Choice items ma y be classified as to the "one right answer' 
or the best answer" type. When an incomplete statement is used care 
must be taken to ensure that it is equivalent to a direct question. 

1 . The one-right-answer type 

In the one- right-answer type, the trainee will be required to 
identify the one correct response listed among a number of 
others that are totally wrong. In writing this type, special 
care must be taken that there is only one correct response in 
the alternatives. Example: 

An auger bit that bores a 1/2- inch hole is stamped with what 
number? 



a. (t 

b. 2 



c . 8 

d. 4 





50 



2. Thejb est- answer t^ ge 



This variation will rea.H^^ 

sponse from a series of u the trainee s to select tho k 
item helno - les of alternatives ect the best re- 

r : — - -3 



a. 

b. 

c. 

d. 



You are fl nishlng . fJne ' S ° 1Utl °"' 

Eire s *- ~ and n °“« • 

1,6 bear to use 1„ repa ir inf JhTJ&E?"" 

Inlay t wlth°the' i 's th S ^ i ° 1 ' shclIa o 

Fill with Jut T klnd ° f Wood 

Dri11 out the bad S p S ot dUSt *""" the Same kind of Wood 

1 ’r^r. 4 - a sc Items: 

1 • Test items v~i 



red blood cells 
1 vmphocy tes 



" ^n ‘different t^sT"* —— - presses this enneept 

Example; Erythrocytes are s i s , 

y s are also known as 

b - sq u nre ^nuc lea ted £ell J° PhlleS 

2 ' ISS ^SS^SUm s t° purpose 

*' W ” at la thb P^pese served by , 

rvnat principle i s exemplif ied by ? 

c - Why is this done 

d - Who, is the most Important reason for ‘ J 

Example: Blood plasma Is administered t 

■* SeV " e »«•- the purpose oV Pa “ ent 8uff orlng from 

f* Preventing bacter^i 4 * 
b - reducing p ain terial infection 
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3 . Test items relating to cause 



a. What is the cause of ? 

b. Under which of the following conditions is this true. . ? 

Example: What is the cause of septic Inflammation? 

a. Trauma or mechanical irritation 

b. The effects of heat and cold 

c. Chemicals from insects or plants 

d. The presence of bacteria 

4 . Test items relating to effect 

a. What is the effect of .....? 

b. If this is done, what will happen ? 

c. Which of the following should be done (to achieve a given 

purpose ? 

Example: What will happen if the electrolyte level of x battery is 

allowed to remain below the tops of the plates? 

a - The battery voltage will be reduced 
b. The plates will expand 

The battery capacity will be reduced 

d. The sediment will rise to the top 

5 . Test items relating to association 

a. What tends to occur in connection (temporal, causal, or 
concomitant association) with ? 

Example: If the metal diaphragm in the air regulator on a dental 

operating uni 1 - , state the position in which the 

valve will re 

a. Partially open c. Halfway closed 

b. Open d. Closed 

6 • Test items relating to recognition of error 

a. Which of the following constitutes an error (with respect 
to a given situation) ? 

Example: What liquid should NEVER be used for cleaning ball or 

roller bearings? 



a. 




d. 

71 

52 



Alcohol 
Mineral oil 



Lard oil 
Vegetable oil 



7. Test items relating to identification of error 



a. What kind of an error is this ? 

b. What is the name of this error ? 

c. What recognized principle is violated . . ? 

Example: While a boat is being refueled, a static spark ignited 

the gasoline fumes and a serious fire resulted. The 
man in charge of the operation might have prevented the 
fire if he had made certain that the nozzle of the 
filling hose had a 

a. hard rubber surface 

b . grounded wire attached 

c. bake 1 it e tip 

d. 40- mesh wire gage protector 

8 * Test itemr relating to evaluation 

a. What is the best evaluation of . . .(for a given purpose) 

and for what reason. . . . ? 

Exam;- ■ ; When the number of cases is small (e.g. less than 20) 
and the magnitude of the values is likewise small, the 
use of an assumed mean in the computation of the mean 
can best be evaluated as 

a. less efficient than computation from actual values 

b. likely to distort the value obtained by the intro- 
duction of a constant error 

c. more accurate than the actual values 

d. neither better nor worse than computation by other 
meH.ods 

9* Test items relating to difference 

a. What is the (or an) important difference between a plain 
milling cutter and a spiral milling cutter? 

a. provides a smoother cutting action 

b. provides for more chip clearance 

c. provides a side thrust to relieve the load 

on the bearing 

d. make grinding easier 
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10 . Test items relating to similarity 



a* What is the (or an) important similarity between . . . ? 
Example: The directional gyro and the artificial horizon utilize a 

a. rigidity in space of a gyro 

b. precession of a gyro 

c. high sp^eed of a gyro 

d. dynamic stability of a gyro 



1 1 • Test items relating to arrangement 

a. In the proper order (to achieve a, given purpose or to follow 
a given rule), which of the following comes f**rst (or last, 
or follows a given item) . . . ? 

Example: In the event of a casualty to the lube oil heater, the 

first step to be taken is to 

a. recirculate the 'oil 

b. secure the lube oil pump 

c. cut in the lube oil storage tank 
_d. secure the lube oil heater 



12 . Test items relating to incomplete arrangements 

a. Before starting a grinding operation on the surface grinder, 
the worker should tnnke a last check to see that 

a. the wheel is properly mounted 

b. the wheel is properly dressed 

c. the work is properly aligned 

d. the magnetic switch is turned on 



13 . Test items relating to a common principle 

a. All except one of the following items - micrometer, vernier, 
caliper, gear tooth caliper, hermophrodite caliper - are 
properly classified as numerical measuring instruments. From 
the following, the one which can be substituted in the above 
series is: 

ji. depth micrometer d. surface gage 

b. sine bar e. dividers 
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14. Test itema relating to controversial subjects 

Although not everyone agreed with the desirability to change the 
national holidays for our national heroes from their birthd&te to 
a specific Monday during the month of their birth, those who 
supported this change did so primarily for the reason that this 
arrangement; 

a. is better for business 

b. eliminates absenteeism 

£. provides an extended weekend 
d. is good for public morale 

15 . Test it ems requiring the most inclusive answer 



Example: Which of the following 

a. smallpox 

b. tuberculosis 



terms is the most inclusive ? 

c. scurvy 

d. disease 



16. 



items requiring the most dissimilar answer 



Example: Which of the following terms does not belong with the other 

three ? 

a. shark c. giraffe 

b ■ a P e d. kangaroo 



V. TEST ITKMS IN A NEnAT TT ’b 
Test Items with Several Responses 

A negative form is used for test items that normally have several 
responses which are equally good. If the problem was stated in a posi- 
tive form, the alternatives for such an item would inclide several cor- 
rect responses and one response which would not be correct or would be 
weaker than the others. Since it is not practical to have an item with 
more than one correct response, the item is prepared in a negative form 
The trainee is, thus, asked to select the response that could not have 
been correct or would not have belonged with the other correct response 
if the item had been asked in the positive form. 
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1 . Test items relating to the negative variety 



Examp 1 e 



Positive statement 

Which of the following engine 
lathe accessories operates on the 
principle of circumferential 
pressure : 

a. three- jaw chuck 

b. four-jaw chuck 
£. spring collet 
d. face plate 



Negative statement 

Which of the following engine 
lathe accessories does NOT oper- 
ate on the principle of circum- 
ferential pressure: 

a. three -jaw chuck 

b. four- jaw chuck 

c. spring collet 

d. face plate 



Comment : The item on the left in positive form has three correct 

responses and, for this reason, can not be used. When stated nega- 
tively, as in the item on the right, there is only one correct re- 
sponse . 



VI. DEVELOPING MULTIPLE- CHOICE ITEMS 
A. Rules for Formulating Test Items 

During recent years, individuals engaged iu the construction of good 
multiple-choice test items have developed a series of suggestions or 
rules that have become widely accepted. These suggestions and rules will 
be presented in four groups: 

1. General rules 

2. Kales ft i. developing the acem or statement of the problem 

3. Rules for developing the alternatives or responses 

4. Miscellaneous rules and policies 



B General Rules 



1. The item should be expressed as clearly as possible. Unlike ordinary 
reading material in which extensive context helps to clarify the 
meaning of any particular phrase or sentence, the mult iple- choice 
item must be clear in and of itself. The power to discriminate be- 
tween the better and poorer cczzxt sman, at a particular level, may be 
seriously limited by lack of eziarity. In achievement testing, the 
difficulty of an item should rcLate to the problem involved rather 
than from the words in which -X is expressed. Achievement test items 
should not be verbal puzzles; -hey should indicate whether or not 
students can produce the correct answer to a problem. 




Lack of clarity in a test it^en may arise from many causes. It may 
be due to the inappropriate cioice., and awkward arrangement of words 
or disorganized thinking of ~the "test writer. Many ideas for test 
items are vague and general 4ien first written. Before emerging in 
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final form, they need to go through a critical examination, evalua- 
tion, and revision. By doing this, clarification of ideas will go 
along hand-in-hand with improvement in the wording of a test item. 

The test item, as a whole, should present a problem which, if an- 
swered correctly, will show that the student has achieved or learned 
an important aspect of the course of study in the subject matter 
area being tested. 

The test item should be realistic and practical. it should call for 
knowledge with which the tradesman or industrial technician should 
be familiar. Asking a tradesman or technician about some trivial, 
insignificant, unimportant fact does not give him an opportunity to 
demonstrate his knowledge of the more important aspects of the sub- 
ject matter areas covered by the examination. Problems which can be 
answered by general intelligence or general knowledge should be 
avoided. 



a . Test item relating to practical problems 



Examples 



Not a practical problem p - , problem 

In order to construct a frame, 
two pieces of cypress 1" thick, 
8" wide, and 6* long are needed. 
If the cost of cypress is 24<? 
per board foot, what will the 
lumber cost? 

b. T" x W" x L’ 





12 




o 

• 


T" x W” x 


L" 




144 




d. 


T" x W’ x 


L" 



12 



Comment ; In the item asking for the board foot formula, the student 
is very apt to say, "So what?". It is doubtful that he would have to 
recite the formula for finding board feet. That is all the item would 
measure. It is not too practical. If it is desired to measure the 
ability of the individual to use the formula, this can be accomplished 
much better with the more practical problem which deals with informa- 
tion that may have to be used or known on the job. 



Which of the following is the 
formula to use in determining 
board feet? 

a. T" x W* x L" 

144 



b. 


Test items relating to useful 


information 




Ex amp 


>les 




Useless Information 


Use 


ful information 


How many basic joints are used 
in welding? 

a. five 


It is not advisable to attempt to 
weld cast iron to steel with a 
steel filler rod because 


b. 


two 


a . 


cast iron has a higher melting 
point than steel 


c. 


three 


b. 


steel has a higher melting 
point than cast iron 


d. 


nine 


c . 


neither metal is stable at 
welding temperatures 






d. 


the iron will ruin the struc- 
ture of the steel molecules 



.Comment : The information required to answer the item at the right cor- 

rectly is useful on the job. There is no point in asking the item at the 
left, since a knowledge of how many joints are used in welding is of no 
value as far as doing a welding job is concerned. 

c * Test items involving general versus specific information 

Examples 



General knowledge 



Specific information 



In arc welding the greatest 
q zard is 



To guard against the greatest safety 
hazard in arc welding it is essen- 
tial at all times to 



a , 


burns from hot 


sparks 


a . 




and slag 




b. 


b. 


burn damage to 


the feet 


c . 


c . 


injury to the 


eyes 


d. 


d. 


flash bums 







wear a protective apron 
wear safety shoes 

wear a helmet with proper lens 
ground the work properly 



Comment Both items are supposed to test knowledge of the most serious 
hazard in arc. welding. The answer on the left can easily be guessed. 

More specific knowledge is required to select the correct item on the 
right . 



si . Test ite ms involving appropriate- inappropriate statements 

The problem should be stated in language which is common or appro- 
priate to the job and/or subject matter. It is important to avoid 
copying a problem direcjfcly from a manual or a textbook. Some exam- 
ples of statements of problems inapptT^fft’iately and appropriately 
worded follow. * 1 
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Examples 



Inappropriate statements 

1- What is the function of the 
clapper box on a shaper? 

2 . The main factor which determines 

the speed of operation of the 
acetylene welder is the 

3. What operation must be under- 
taken first in sharpening a 
handsaw? 



More appropriate statements 

1. The clapper box on a shaper 

is used for 

2. The speed of work of the acet 

ylene welder depends mainly 
on the 

3. The first thing to do in 
sharpening a handsaw is to 



e • Test items involving textbook or job language 

Two other examples, taken from achievement tests, are given below: 



Examples 



Textbook language 

Which of the following oscillators is 
employed to take advantage of the 
inter electrode capacity of a triode? 

a. tuned plate - tuned grid 

b. relaxation 

c. Hartley 

d. colpit 



Job language 

Which of the following oscillators 
makes use of the interelectrode 
capacity of a triode? 

a. tuned plate - tuned grid 

b. relaxation 

c. Hartley 

d. colpit 



C ommen t : The item at the left was copied from a manual. It emphasized a 

relatively unimportant point, author ization. The language is awkward and 
tends to make the item difficult to read. The item at the right is worded 
in language which the trainee will use on the job and should understand. 



f . Test items involving clarity of words 

Persons responsible for writing items should choose words that 
have precise meaning whenever possible. Lack of clarity frequently 
arises from inappropriate word choices. Many commonly used words and 
phrases have no precise meaning. Others have no meaning that applies 
accurately in the context in whi^h they appear. 




a, 
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Examp 1 e ; 



In the 1948 presidential campaign, Truman yielded unofficially 
to Dewey on which of the following points? 

a. That the un-American activities investigations were not direct 
attacks on the Democratic Party 

— “ That the tone of the political campaign would be kept on a high 
level ‘ 



c. That the civil rights program be abandoned 

d. That there should be an equal balance of Democrats and Republi- 
cans in Congress to insure sound legislation 



Comment ; 



The words "yielded", "unofficially". 



and "points" are vague. 



S # Test items involving simple sentences 

Personnel writing items should avoid complex or awkward arrange- 
ments. In regard to this, F. L. Ebel makes the following suggestions: 

s - ruc ^ure of sentences used should be as simple as possible. 
It is often advantageous to break up a complex sentence into two or 
more separate sentences, A qualifying phrase should be placed near 
the term it qualifies. In general, it is desirable to make clear the 
focal point of the problem early in the sentence and add qualifications 
or explanations later. Finally, it is often helpful for the item 
writer to ask himself, 'Just what is the point of this item?'. In the 
answer to this question, an item writer may find a simpler, more direct 
wording; or, he may discover that it has little point; or one that is 
not worth testing." 



h * Test Items must be Independent of each other 

The items in an examination should be independent of each other. 
The information contained in one item should not give away the answer 
to another item. 



Examp 1 e s 



When a naval vessel i s at anchor 
or moored, it flies the National 
Ensign at the 

a . ma inma s t 

b. foremast 

c. jackstaff 

d. flagstaff 



When a ship is not underway, the 
National Ensign should be dis- 
played from the flagstaff from 

& . 0800 to sunset 

b. sunrise to 1830 
c m 0800 to 1830 
d. sunrise to sunset 



Comment : It: is obvious that the statement of the problem in the item at the 

right gives away the answer to the item at the left . 
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Judging the Difficulty Level of Questions 



The level of difficulty of the item should be adapted to the 
group and purpose for which the item is intended. While subjective 
judgements of item difficulty are not too accurate, personnel who are 
well acquainted with the general ability and knowledge of the competen- 
cy level and who know something about typical performances on similar 
and old items can do a useful job of judging item difficulty. 



5 . P itfalls in Judging Item Difficulties 

In connection with the adjustment of item difficulty, two pitfalls 
must be avoided. The first Is the application of a ’’minimum essentials" 
concept to an achievement test. According to this concept, a test 
should Include only those Items that all journeymen should, according 
to minimum essential standards, be able to answer correctly. A test 
consisting of items meeting this requirement will, in all probability, 
be too easy. 

The second pitfall (or mistake) is writing (or selecting) Items 
frcm the standpoint of what the Ideal journeyman should know rather 
than in terms of what the typical tradesman knows. A person who is un- 
realistic about the typical tradesman is likely to construct or select 
unreasonably difficult test items. Some people believe that such a 
test writer has high stan da rds. I t wo u ld be mor e appropriate to say 
that such a person is unrealistic . — 

a . Test items involving increasing difficulties 

It is desirable, and often necessary, to sample an area of know- 
ledge at different levels of difficulty. The following example 
illustrates how items of increasing difficulty can be written to 
cover the same subject matter. 
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Example 

Easy: Which of the following should be used to extinguish a fire In a 

small electric motor? 



a. A bucket of sand 

b. A stream of water 

c^. A portable fire extinguisher 
d. A blanket or a mattress 



More difficult: What type of fire extinguisher is the best for extin- 

guishing a fire in an electric motor? 



a . 
b. 



Foam 

c0 2 

Pyrene 

Carbon Tetraohlor ide 







c . 

d. 



More difficult: Even though a pyrene extinguisher is almost as effect- 

ive as a C0 2 extinguisher for extinguishing a fire in 
an electric motor, it should NEVER be used due to the 
fact pyrene will 

!!■• form a toxic gas when contacting hot metal 

b. conduct electric current 

c. render motor unfit for repair or salvage 

d. explode when near an electric spark 

Most difficult: What is the primary extinguishing action that takes 

plf.ce when a C0 2 extinguisher is used on a fire in an 
electric motor? 

a. The heat of the fire causes the C0 2 to expand 
which thins out the oxygen below the amount 
needed for combustion. 

b. The C0 2 being extremely cold, quickly cools 
every part of the motor below its flash point. 

c. The CO 2 because of its great density, settles 
in a blanket, displacing the air and oxygen 
necessary for combustion. 

d. The CC >2 combines chemically with the air to 

form which effectively removes oxygen 

from the fire area. 

VII. THE CONTROL OF DIFFICULTY LEVEL FOR TEST ITEMS 
Ways of Changing the Difficulty of a Test Item 

There are two ways of changing -the difficulty of a test item; (1) 
change the statement of the problem, or (2) change the alternatives. 
Changing item difficulty by changing the alternatives is quite limited in 
the practical situation. The statement of the problem usually determines 
the range of possible alternatives. Alternatives which are altered toward 
increased complexity and, hence, difficulty, would probably necessitate a 
more difficult or complex statement of the problem. 

It should be apparent, then, that changing item difficulty by chang- 
ing the statement of the problem is probably the better approach. It is 
much easier and more satisfactory to change the statement of the problem 
and find suitable alternatives, especially the distractors, to the state- 
ment of the problem. It is important to remember that distractors should 
do just what their name implies. They should distract the less qualified 
students from the correct response. Distractors can do this only if they 

are plausible answers Items sometimes appear easier because distractors 
fail to distract. 
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Check Points for Evaluating Questions 



Can you answer "yes" to each of the following questions about each 
of the test items you have written? 

Is the test item: 

1. stated as clearly as possible, 

2. realistic and practical, 

3. dealing with an important aspect and knowledge of the job, 

A. phrased in the language of the job, 

5. stated in words having precise meaning, 

6. presenting an independent problem, 

7. of appropriate difficulty? 

If you can answer "yes’ 1 to each of the above questions concerning 
test items which you have written or revised, then your items should be 
specific in terms of content and purpose. 



VIII . RULES FOR DEVELOPING THE STEM OF THE TEST ITEM 
Test Items Inv olving a Clear Central Problem 

The statement of the problem should be or contain a clear central 
issue to be solved. If a well-informed student can give the correct an- 
swer after reading the statement of the problem (which in this case is 
written as an incomplete statement) without reading the alternatives, it 
can be assumed that a clear central problem has been stated. 



Examp les 



No Central Problem 

The persistency of a gas 

a. refers to its toxic effects 

b. refers to its identifying odor 
£. is the duration of its effect- 
iveness after release 

d. is its clinging effect on the 
skin 



Clear Central Problem 

The persistency of a gas refers 
to its 

a. toxic effects 

b . identifying color 

_c* duration of effectiveness 
after release 

d. clinging effect on skin 



Comment s : The stem of the problem at the left (the persistency of a gas) 

is really not a problem at all because it has no central thought. The 
candidate does not know, from the stem, what problem is to be solved. The 
item can be improved by bringing the central problem into the item stem 
as illustrated at the right. 



Test Items Involving Pure Memory vs. Understanding and Principle s 



When developing the statements of the problem try, as much as possi- 
ble, to avoid a problem which requires pure memory for its solution. A 
candidate may know words, or phrases, without knowing their meaning or 
application. Items involving only memory for solutions will probably 
not tell which of the candidates have gained the greatest mastery of the 
subject matter. Items which call for an understanding and application 
of facts and principles are those most apt to tell what needs to be known 
about each candidate’s competency. Candidates who are most successful on 
the job as craftsmen are those who are able to apply to job situations 
facts and principles which they havo learned. 

T est Items Must Be Stated Briefly and Completely 

The problem should be stated clearly, briefly, and completely. It 
should be stated in the fewest possible words, but it should include all 
the essential information in relation to the alternatives. 



Example 

Lack of information Complete information 

An external thread would be A hardened, threaded section of a 

work piece is machined to a pre- 
cise micro-finish by 



a . 


turned 


a. 


turning on a lathe 


b. 


cut with a die 


b. 


cutting with a screw threading 
die 


c_ . 


ground 


c. 


grinding or using a precision 


d. 


tapped 




grinder attachment 






d. 


tapp ing 



Comment: The item at the left does not give sufficient information upon 

which to base an intelligent answer. There is no problem and there is no 
single answer as there are the three underlined answers. Additional in- 
formation in the item at the right makes the problem clear and a single 
answer possible. B 

The Problem Sho uld Contain Only Material Applicable to its Solution . 

Information which is not necessary for obtaining an answer to the pro- 
blem should be omitted. Such information may tend to destroy the validity 
of the item as a measure of what it is intended to measure. Since stu- 
dents vary greatly in reading ability, it is undesirable for reading abili- 
ty to be a factor in determining the correct response. In general, items 
should be kept as short as is consistent with clear statements. Some in- 
troductory statements that are not, strictly necessary may occasionally be 
justified if they help to clarify* five problem or to establish its impor- 



Examp 1 e 



Irrelevant information 

The electronic serviceman must be 
able to identify certain causes of 
difficulties in receivers by their 
sound. If yr— hear a high pitched 
squeal in the aeput of an AM re- 
ceiver, this defect is usually 
caused by 

a. excessive amplification 

b. poor tracking of the local 

oscillator 

c. a defective output trans- 

former 

d. a defective detector 



Relevant information 

A high pitched squeal in the output 
of an AM receiver is usually caused 
by 



a. excessive amplification 

b. poor tracking of the local 

osci 1 la tor 

c. a defective output trans- 

former 

d. a defective detector 



Comment : The information contained in the item at the left is not neces- 
sarily to present the problem. The irrelevant information requires addi- 
tional reading time of the trainee and makes it more difficult for him to 
determine the nature of the problem. 



E ’ Tj) e Ability to Sele ct Material From the Problem Which is Applicable to 
bne Correct Solution is Often Quite Importan t 



Examp 1 e 



What is the amount of net 

Closing inventory 

Purchases 

Sales 

Opening inventory 



increase inventory in 

$ 91,000 
$ 80,000 
$ 62,000 
$ 58,000 



the following situation? 



a. $149,000 

b. $ 51,000 

c. $ 33,000 

d. $ 15,000 



Comment: In the preceding problem, a person must not only be able to 

perform the arithmetic involved but he must be able to select the 
relevant information from the problem. Irrelevant information in the 
problem (purchases and sales), which is common in a practical solution, 
is necessary in order to test the trainee's ability to select the infor- 
mation he needs to arrive at the correct answer, 
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F . The Problem Should be Stated in Positive Form if Possible 

Students may respond to an item in the negative form as if it were 
stated in the positive form. Capitalizing words such as NOT, POOREST, 

NO, LEAST, NEVER, etc. may help to overcome this to some extent. If 
these words, however, appear in the statement of the problem and they 
neither add nor detract from th lesired emphasis;, they should, natura 1 ly , 
be in the lower case. A problem usually has more significance and is a 
better problem if it is stated positively 



G. When Writing Reasoning Problems in Related Mathematics, Science, Drawing 
Interpretation, Avoid Making the Computational Parts of the Probl em 
Unintentionally Difficult 

Trainees frequently answer reasoning problems in mathematics, science 
and other related subjects incorrectly, even though they have reasoned 
correctly. This is due to a slight error in computation, and not in rea- 
soning. The person writing the item should decide whether he wishes to 
test computational skill involving complex or time-consuming calculations, 
or an understanding of mathematical principles. In this latter case, the 
test item should keep computational difficulty to a minimum. 

li . The Stem Should Include All Qualifying Information Needed t o Provide a 
Reasonable Basis for Response Selection 

Sometimes, a person writing an item does not state explicity all of 
the qualifying information. A person taking an examination must have all 
qua 1 if ic:at ions specif ica 1 ly stated. 



Example 

If a ship is wrecked in very deep water, how far will it sink? 

a. just under the surface 

b . to the bottom 

c. to a depth where the pressure is equal to its weight 

d. to a depth which depends in part upon the amount of air 

remaining in the ship 

C omment : : A number of capable people would probably select response 

number d, instead of the desired correct response b, because they consid- 
ered the possibility (which the person that wrote the item failed to ex- 
clude) that a wrecked ship might not sink completely but might remain 
partly submerged. In that case, response d, while not too good, is the 
best response available. 
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c . 



Hh en a Stgm Contains Lengthy o r Involved Oualifvine TnfnrmpUnn t- .t „ 

the s5jugSOfe i ^|ig F^^nd 

S t a t e ment of the Problem Reques ting the Des IretTT^ f o-K-maH ^ 



It is, sometimes, 
as a complete sentence 
or implied question, 
oping an item stem: 



clearer to state the qualifying 
, and then use another sentence 
The following suggestions may b 



e 



information first, 
to state the problem 
helpful when devel- 



1 . 



If there is no qualifying information, 
first (who, what, where, when, etc.). 



state the question 



2 . 



If there is one qualifying clause, 
this makes the problem clearer. 



state the question last if 



3 . 



If there are two or more qualifying clauses 
sentence and state the question in the last 



use more than one 
sentence. 



It is well to remember that clarity is the main consideration 
Clarity of Statement-- 



^Qualifying information last 

What safety precautions must be 
observed first when dressing or 
finishing the edge of a board 
on the jointer? 



Qualifying information first 

When dressing or finishing the 
edge of a board on a jointer, 
what safety precautions should 
be observed first? 

as in the 
based is known 
the left, 
the question is 
are more quick- 



Comment: When the qualifying information is stated first 

stem on the right, the situation upon which the question i 
- the person - ta king the test, sooner than in the stem at 
"b- thB necessary information is identified first (before 

ly grlsped! Bntire Pr ° blem is ^arer and the requirements 



IX. RULES FOR DEVELOPING THE ALTERNATIVES 
The Correct Response Must Be Right 



riKht * 10 ThP^f resp °nse , or the best response, should be unquestionably 
^ H re f erence from which the information for the item is taken 

s^on a : a r?r^ tly acCepted as authoritative. In order to a vo id coSu- 
, * r p ally correct response should not be used even if it i « t-h 

vJltlSf It ft‘rr ‘‘“'-natives. ttls can , bl r Jlji d b y «- 

" Wh a t^as^ the ‘purpose of ^ f ° £ 

was to ... # . 



; tewrite it to ask, ’’One purpose of 
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B. 



— P^- s t ractor s (wrong responses) Sr 
Which A re Commonly Made By Less Qua 

Such errors are usually due to 
errors may be used as distractors w: 
qualified and the le .a qualifier earn 
by people, in genera; should net be 
tions are usually due to ignorati e ar 
teaching/ learning situation. 



ild Be Representative o f Errors 
ied Tradesmen 

ors in technical judgement. These 
^ discriminate between the better 
dates. Popular misconceptions held 
sed as distractors. such misconcep 
should have been corrected in the 



C. 



Ea c h_ of the Alternatives should Liter;: Hv a Possible Answer 

The distractors should be closely f t-n nrn ti 

the poorer trainees, but not any lar^e Pe^nt^f Ihe'bett^ nuaU^ef 

nlausibl USS these , as answers. The distractors should be important 
plausible answers. They should not be obvious errors. important, 



Example 



jjB£ elated wrong answers 

One of the requirements for ad- 
vancement in grade in the Civil 
Service is that the candidate 
mus t 



Plausible wrong answers 

One of the requirements for advance- 
ment in grade in the Civil Service 
is that the candidate must 



a. be working in the department 

b. be married and have a family 

pass a qualifying examina- 
t ion 

d. be recommended by his 
superior 



a - have at least three months 

experience in the higher grade 

b. be- recommended by the personnel 
office 

— * pass a qualifying examination 

^ * have at least two years exper- 
ience in the work of the 
higher grade 



Commen t: 
likely to 
ject than 



The distractor 
be used by the 
the unrelated 



S, as plausible wrong answers, are much more 
person who does not know anything about the sub- 
distractors which are obviously wrong. 



D. 



— 1 — — the Distr a ctors Should Be Appropriate 



to the Item Stem 



Example 

Why do living organisms need oxygen? 




a. Purification of the blood 

b. Oxidation 

c. Release of energy 

d. Assimilation of the blood 
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Comment : 
stem for 
stem . 



In this item the alternatives are not appropriate to the item 
the responses are not stated as reasons which are required by the 



E. 



The Item Writers Should Avoid 
Correct Response 



•^• rre ^ evan t Clues Which May Give Away the 



If a number of candidates who would not 
correct response notice the clue and respond 
itely weakened in that it measures something 
was intended. 



normally be able to 
correctly, the item 
other than that for 



choose the 
is defin- 
which it 



1 - ^ - releva — t - clues exi st whenever there are any syst ematic formal 
d ifferences between the c o rrect response and the distractor a . 

A good example would be when there is a tendency to place the 
answer in one favored position among the several alternatives If 
a trainee should observe that the third response is most frequent 
ly the correct answer, or that the fourth response is seldom cor- 
rect, he may use such observations as a foundation for successful 
guesses on other items. Test developers, responsible for writing 
items or for assembling examinations, should be particularly 
alert not to favor any one number as the correct response, and to 

attempt to get a good distribution among all the numbers for cor- 
rect response. 

2 * . Probably one of the mo st common types of i rrelevant clues 

involves common element* in the item stem and~in the answer. 

An example of an item having common elements and a reworked 
or corrected item is given. 



Common element 



Example 



No clue 



The divisor is a term used in 

a. addition 

b. subtraction 
_c. division 

d. multiplication 



What arithmetical method should 
be used to find the number of 
square feet of floor space in a 
room which is 12 feet wide and 
24 feet long? 

a. division 

b. multiplication 

c. subtraction 

d. addition 



in the item at the left there is an element which is common to 



The word "divisor 11 in the 



pro- 
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Comment : 

both the problem and the correct response. ^ uj_ V j_ 

blem gives away the answer "division" in the alternative!" This is so ob 
vlous that most any untrained person could get the correct answer. When 

the'correct methods is necessary to find 

69 



"never 11 , and "always" 



are 



3 . 



W ords like ’’all 11 » "more 91 , "certainly* 1 
called specific determiners. 



The use of specific determiners tends to provide irrelevent 
clues to the correct response because statements including them 
(unless great care is taken) are predominantly false. Similarly, 
modifiers such as "sometimes", "generally", and "usually" should 5 
be used in both correct and incorrect alternatives, if they are 
used at a'll, since they, otherwise, may give a clue to the cor- 
rect answer. 



4. The length of the correct response should not be a clue to the 
correct response. 

If the correct response Is consistently either longer or 
shorter than the other alternatives, it is possible that the 
trainees will discover this and rely on this tendency rather 
than on knowledge of the subject matter in the examination. 

5 . Clues due to grammatically incorrect sentences. 

A clue to the correct response or to the elimination of some 
distractors is sometimes due to the fact that the statement of 
the problem does not make a grammatically correct sentence in 
combination with each alternative. This should be avoided. 



Examples 



Grammat ical ly incorrtqt 

On a road map, a main highway 
is shown by a 

a. single heavy line 

b. two red dotted lines 

c. parallel broken lines 

d. parallel unbroken lines 



Grammatically correct 

On a road map, a main highway 
is shown by a 

a. . single heavy line 

b. pair of dotted lines 

c. set of parallel broken lines 

d. set of parallel unbroken 

1 ines 



Another example which is very common follows: 



When filing metal, the file should 
be cleaned frequently with a 



a. 


wire 


brush 


b. 


oily 


rag 


c. 


emery 


board 


d. 


file 


card 



When filing metal, the file 
should be cleaned frequently 
with a/an 

a. wire brush 

b. oil rag 

c. emery board 

d. file card 

since alterna- 
that there are 
remedied, in the 



Comment : In the item at the left, it should be obvious that 

tives (b) and (c) make the problem grammatically incorrect 
only two possible answers, (a) and (d) . This situation is 
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item at the right, by adding "an" along with "a" at the end of the 
problem. Each alternative on the right makes a grammatically correct 
sentence and the possibility of guessing the correct answer is reduced. 

6 ‘ Al ternatives should deal with similar materials or elements 

The alternatives should, usually, deal with similar ideas, 
materials, or elements which are expressed in parallel form. 

Examp 1 e 



Unrelated alternatives 

In the Civ.il Seirvica a 

beginning typist must: 

a. be able to type from copy 
at the rate of 50 net words 
per minute 

b. set up and maintain confi- 
dential files 

c. dust the desk 

d. be punctual 



Comment In the item at the left 
similarity of material confuses 
that alternative (a) was 
at the right deals with 
alternative is in terms 



Similar alternatives 

According to the statement of 
Qualifications of the Civil Service 
a candidate for beginning typist 
must be able to type from copy at 
the rate of, at least, how many 
net words per minute? 

a . 30 

b. 35 

c . AO 

d. 50 

and 
clue 

The item 
Each 



the lack of parallel structure 
the whole problem and provides a 
probably intended for the best answer, 
similar material and the problem is clear, 
of net words per minute. 



^ * JL r re l e vant clues may be constructive on rare occasions 

It should be noted that irrelevant clues may, on rare occasions, 
be used in a constructive manner. The deliberate planting of 
some of the clues mentioned above may tend to defeat the rote- 
learner or to make the distractors highly attractive to those 
trainees whose knowledge is superficial. 



Each Response Must Make Grammatical S 



ense 



Alternatives are, frequently, made unnecessarily long by including 
Ln each alternative material which should be included in the stem. The 
stem should, in general, include any words that would otherwise have to 
be repeated in each response. it should be noted, however, that it is 
not always desirable to eliminate all words common to the responses. 
Sometimes, it is necessary to introduce each response with a word or 
p rase to make grammatical sense. Also, if the retention of common words 
rn aii the responses makes the item easier to understand, they may be 

retained. Usually, however, the item writer will want to transfer common 
"*)rds to the stem. 
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Material repeated in each 
alternative 



Example 



A portable electric submersible 
pump with a clean strainer that is 
pumping aga ist a 50- foot head or 
less will discharge. 

a. 100 gallons of water per minute 

h. 140 gallons of water per minute 

c. 160 gallons of water per minute 

jd. 180 gallons of water per minute 



Material taken from alternatives 
and placed in the problem 

How many gallons of water per 
minute will be discharged by n 
portable electric submersible pump 3 
with a clean strainer, that is 
pumping against a 50- foot head or 
less? 

a. 100 c. 160 

b. 140 d. 180 



Comment ; : The problem at the right is more 
the one at the left because material which 
repeated in the alternatives. 



easily read and understood than 
is a part of the problem is not 



Responses That Overlap Or Include Each Other Should Be Avoided 



Example 



What percent of the total loss, due to hail, is the loss to growing crops 

a. Less than 20% c. More than 50% 

b. Less than 25% d . More than 55% 



jg omment : This item is actually a two-response item. The choice is between 

(a) and (c) for, if ( a ) is correct, then (b) is also correct- if (d) is 
correct , (c) is also correct. 



Acceptable Alternatives 



"None of the above", in some items, is an acceptable alterna 
l or . example, the subject of safety precautions is an area 
in which it is sometimes necessary to find out whether or not the 
individual would not, under any circumstances, commit the act pre 
sen ted in any of the alternatives. Another use for "None of the 
aoove is as a distractor when the correct response seems to give 

itself away and would be too easy to see in a set of ordinarv al- 
ternatives. J 



o 



None of the above" is sometimes used in mathematical pro- 
blems as either the correct response or as a distractor. In all 

U u Bd n S thB laSt alternative - Some people feel that 
None of the above should be used as a response only in items 

which are of the best — answer type. There are a few limitations 
n connection with usi^jg- it as an alternative. It SHOULD NOT be 
used if the statement of the problem is an incomplete statement. 
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A pattern should not be developed in using it, and it should be 
used about as frequently as a correct response as it is as an 
incorrect response. It should not be used just to n fill in" when 
a final distractor is difficult to develop. 



*A11 of the above" , in some items, is an acceptable alterna- 
tive. There is some difference of opinion, however, concerning 
its proper use. The same discretion should be followed when us- 
ing "411 of the above" as with the use of "None of the above". 

In all instances, it must be used as the last alternative. It, 
too, should not be used as an alternative following the statement 
of a problem which is an incomplete statement. 



"Any of the above" will not be used as an alternative. 



Items requiring negative alternatives should be used as 
little as possible. Items with positive alternatives may be 
clearer to the candidates. The nature of the subject matter usu- 
ally determines the use of this type of alternative. The follow- 
ing examples are given to illustrate this type of item: 



Examples 



Negative alternatives 

When refilling water glasses 
during a meal, you should 



a. remove the glass from the table 

b. always place the glass on a 

serving tray 

£• not move the glass from its 
position on the table 
d, never move the glass more 

than six inches to the right 



Positive alternatives 

What should you do with a water 
glass when refilling it during 
a meal? 

a. remove it from the table 

b. place it on a serving tray 
.£. leave it in its position on 

the table 

d. move it six inches to the 
right 



Comment; In the item at the left, alternatives (c) and (d) are stated 
negatively. In the item at the right, all of the alternatives are stated 
positively. 



Instead of repeating two alternatives, in combination to 
form a third alternative, variety is added to the Item by using 
the alternative numbers instead of repeating the alternatives 
themselves . 
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Example s 



Repeating alternatives 

What measure(s) of central 
tendency should be used when 
determining the score which 
occurred most often among a 
set of examination scores? 



a * Mean 

b. Mode 

c. Mean and mode 

d. Median 



Using alternative numbers 

What measure (s) of central 
tendency should be used when 
determining the score which 
occurred most frequently 
among a set of examination 
scores? 

a . Mean 

b. Mode 

c. Both (a) and (b) above 

d. Median 



What will most likely cause an 
individual to obtain a different 
score on the same test when 
taking it a second time? 

a . Practice effect 
b* Additional study 

Practice effect and 
additional study 
d # None of the above 



What will most likely cause an 
individual to obtain a different 
score on the same test when 
taking it a second time? 

a. Practice effect 

b. Additional study 

_c. Both (a) and (b) above 
d. None of the above 



X. MISCELLANEOUS RULES AND POLICIES 



A . Suggestions Pertaining to Numbers 



The following suggestions pertain to the handling of numbers in the 
writing of multiple-choice items: 



1. It is suggested that the numbers from one to ten be spelled out, 
except in the following situations: 

a. In mathematical problems. It Is common practice to use 
numerals (figures) . 

b. When a number greater than ten appears in the statement of 
the problem, or question, or in the alternatives, or both, 
all of the numbers used in the item should appear as num- 
erals . 



O 
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c. 



Fractions appearing in mathematical problems, which are 
accompanied by a whole number, should appear as mixed num- 
bers regardless of whether or not the whole number Is less 
than ten* 

Decimals should always appear as Arabic numerals. 
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2 . 



Fractions, appearing alone as alternatives, should be arranged 
neatly in one of the following patterns; 



a. 1/2 

b. 3/4 

c. 5/6 

d. 7/8 



3. Numbers should 



a . 




1/2 


L. 

L J „ 


3 


1/4 


c . 


9 


1/2 


d. 


9 


3/4 



appear in ascending or 



a . 


2 


1/4 


b. 


8 


1/4 


c . 


76 


1/4 


d. 


108 


1/4 



spending order. 



Example 



Descending order 



Ascending order 



a. 40 c. 20 

b. 30 d. 10 



a. 10 c. 30 

b. 20 d. 40 



4. When the numbers one through four stand alone as alternatives, 
or when the numbers 1, 2, 3, or 4 appear as alternatives, they 
should be placed beside their corresponding digit as follows; 



a. 


One 


a. 


1 


b. 


Two 


b. 


2 


c . 


Three 


c . 


3 


d. 


Four 


d. 


4 



5. When decimals are used, the decimal points should be in line. 

Example 



a. 12.34 

b. 123.45 

c. 1,234.56 

d. 12,345.67 



6. Monetary amounts should appear as follows; 



a . $ 2.50 

b. $ 20.00 

c. $ 200.50 

d. $2000.00 



B - Rules Pertaining to Capitalization 

The roliowing rules, pertaining to capitalization, should be ob- 
served in the preparation of test items: 

1* The firs t letter of the first word of each problem should be 
capitalized. 



a . 

b. 

c . 

d. 



$0.25 

$0.55 

$0.65 

$1.50 



a. 

b. 

c. 

d. 



$ 5.50 
$ 50.50 
$ 75.15 
$ 100.00 
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2 . 



Whenever the stem of an item 
the first letter of the first 
capitalized. 



is in the form of a direct 
word of each alternative 



question 
should be 



3 . 



Whenever the stem of an item is in *-i.~ 

statement, the first i -•*-*- Pr n f m, r,‘ c xortn °f an incomplete 
tive should not be capitalized ® fir8 5 word of each alterna- 
would ordinarily be capitalized ” is a P ro P er name and 

in the alternative c< ‘ i>itallzdd «gerdle s s of where its position is 



4 . 



The first letter 
names should be 



° ^ all proper names and 
capitalized. 



abbreviations 



of proper 



C. 



Punctuation A fter Alternatives 

No punctuation is necessarv _f <. 0 .. 

fers to periods after each alternative dlnary alternatives 



This re- 



A. 



VARIATIONS IN MULTIPLE- CHOICE TEST ITEMS 



Test — I terns Involving Multiple ComE letion 



type, °one h or "more Zy^roplrly "comp^e"^^ 8 ° ffe " ed an item of this 
answered correctly only when Ihe appropriate' !1 (premise > • T he item is 
of responses has been selected. Thus re f ponse or combination 

answer only by committing himself as to th,» eXa ? 066 C3n rdach a correct 
of each one of the three or more possible ° r irreleva nce, 

cess requires judgment in rela^ng vlJioSs ™ ThiS Selectio " pro- 
and depends, especially, upon the mmni *. COnsldera tions to the whole, 
ledge and experience with ° f ‘ he candidate's know-’ 

equal S importance 0 and n plausibility^ 0nS of'* 

Of the kind of decision which the leaf 
Application of Multlele Completion Test Item 

ever a problem Involves!’* made ° f th * ,nulcl P le completion technique wher- 



1 . 



a situation which 
ble consequences. 



may require a consideration 



of 



several 



poss i- 






water solution of sodi^hlSrtdSr^i^w^ir^eilll^S a h ^° ncentr ated 



c . 



evolution of chlorine gas 

nl?“lnv 0 o I* 1 !° dlu,n hydroxide solution 
pla.ing out: of metallic sodium on the gqthode 



(a. b) 

correct 



e<5 



cr &} 



2 . 



a situation where one or more conditions must be specified in 
order to define it adequately, 



Example 

Isotopes of a given element are related in that they have the same 

a. number of neutrons in the nucleus 

b. position in the periodic table 

c. arrangement of planetary electrons 

(b, c) correct 



3. two situations which are alike (or unlike) with regard to one 
or more points o£ comparison. 



Examp 1 e 

In both meiosis and mitosis, there normally occurs 

a. polar body formation 

b. homologous chromosome pairing 

c. cleavage of the cytoplasm 

(c) correct 



4. a principle which may be applicable to one or more situations , 



E xample 

Joule 3 are units by which one can express 

a . work 
b • energy 
c . momentum 

(a, b) correct 

5. several considerations which may bear upon a single result 

Example 

The rate of a chemical reaction may be influenced by 

. temperature 
. catalysts 
c. pressure 
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(a, b, c) correct 



c. 



Appearance (Structure) of the Multiple Completion Test Item Type 



Direct? 3ns : For each of the test items (1 through 5) , ONE or MORE 

of the responses given are correct. Decide which response(s) is 
(are) correct and on the answer sheet blacken the correct space. 

1. if a, b, and c are correct; 

2. if only a and b are correct; 

3. if only b and c are correct; 

4. if only a is correct; 

5. if only c is correct; 

MARK ONE SPACE ONLY ON YOUR ANSWER SHEET FOR EACH QUESTION 





Figure 


1. Directions Summarized 




Blacken Space 


1 


2 


3 


4 


5 


Fox' Correct 


a , b , c 


a , b 


b, c 


a 


c 


Responses 




only 


only 


only 


only 



D . Hints in the Construction of a Multiple Completion Item 

1. Each of the choices should be able to stand alone without de- 
pending upon (or excluding) any other choice. 



Examp 1 e 



In studying the rate of oxygen production by photosynthesis during a 
laboratory experiment; it is necessary to 

Good Poor 



a. regulate the flow of CO^ 

b. control the light intensity 

c. maintain constant temperature 



a. generate CO^ 

b. provide a flow of CO^ 

c. measure the flow of CO 

2 
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2 . 



Make the choices refer to as many as possible 
factors which relate to a problem. (A choice 
be stated falsely to serve as a distractor.) 



of the pertinent 
can occasionally 



Example 

The administration of adrenaline may lead to 



Good 

a* bronchiole contraction 

b. increased blood pressure 

c. increased blood sugar 

(b , c) correct 



Poor 

a. increased blood pressure 

b. a harder working heart 

c. increased photosynthesis 



3. In the stem avoid giving any clue as to how many of the 
choices may be correct. 



Examp 1 e 

GOOD: Hybrid sterility may be caused by (followed by multi- 

pie responses) 

POOR: The genetic factors in hybrid sterility are (followed by 

multiple responses) 



4 ' should^e "designated^ 16 «<=*> <=°«e=t choice 

5. Never have an item with no correct choice; the directions 
assert that at least one choice must be correct. 



3 
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XII. CLASSIFICATION ITEMS 
Test Items with a Number of Key Topics 

or Drincinl^o 31 ^?^ 10 ^ J tem " ty P‘ ? Presents key topics, usually concepts 
with or re? ; J°J lowed b 7 a set of phrases. Each phrase is associated 
with, or related to, one of the topics. This item-type provides a auick 

of a !he C £oo? ermS ° f tes ; lng 5 ime) of determining the candidate's mastery 
t£> P ics Presented. The degree of mastery required of the candi- 

in ?ramin?Thr . P ?I! dent UPOn the ski11 raanifa sted by the item-writer 
ceot or ofi^c? ? set may merely probe familiarity with the con- 

di^t.'s P abi?^r eS T-rK° n thC ° ther hand * thG set may probe the can " 

didat _ s ability to apply the concept or principle to a novel situation. 
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It is quite important that the set be so constructed that the key 
topics have a somewhat reasonable relationship to each other in or- 
der to insure that the examinee is required to exercise an appropri- 
ate amount of discrimination. 

Examples of Classification Items 

Directions: Each set of test items consists of five laws or princi- 

ples followed by a list of words or phrases which are numbered. For 
each law or principle select the one principle which is most closely 
related to it . ^ Blacken the corresponding lettered space on the an- 
swer sheet to indicate the correct corresponding law or principle. 

One law or pi'inciple may be used once, more than once, or not at all. 



(A) 


Set I 

Law or Principle 
Boyle ' s law 


i. 


Condition 

Is fundamental to the operation of a 


(B) 


Charles' law 
(Gay-Lussac's law) 


2. 


spray atomizer 

Accounts for the apparent loss of 


(C) 


Archimedes' principle 




weight by an object when it is sub- 
merged 


(D) 


Pascal ' s law 


3. 


Explains the operation of hydraulic 


(E) 


Bernoulli's principle 




brakes 



4, Explains why the canvas tops of con- 
vertible automobiles bulge upwards 
while the car is moving 



5. Can be used to explain why an in- 
flated toy balloon gets bigger when 
heated 

6 * Accounts for the fact that an air 
bubble gets larger as it rises to 
the surface of a lake 



In conditions :1 through 6, the examinee's understanding of each law is 
tested, not by asking him to state the law, but by establishing whether 
he can use it to explain ordinary phenomena. 

Other sets of classification test items would be included on such a test 



XIII. FIVE-RESPONSE MULTIPLE- CHOICE TEST ITEMS 

The Te st Item Type Most Widely Used in Objective Tests is the Simple 
Five-Response Mult iple- Choice Completion Type 

This item type is so familiar to most students that they handle test 
items with little communication difficulty. In addition to this advan- 
tage, broad experience ir many types of test programs has shown this 



item type to be a consistently good one in discriminating degree of 
achievement and ability from the less able students. in writing this 
type of test item, the material should be searching, demanding thought 
rather than the mere recognition of rote-recall answers. 



B * Appropriateness of Five-Rfesponse Multiple-Choice Completion Item Type 

The problem is framed in the form either of an incomplete statement 
or of a question. Four examples are used to show different applications. 

Directions: Each of the questions, or incomplete statements, is followed 

by five suggested answers, or completions. Select the one which is best 
in each case. Blacken the corresponding lettered space on the answer 
sheet . 



1. When the problem presented has a unique solution 

E xample : A cell having a flat, disc-shaped construction can exchange 

a given quantity of gas more quickly than a spherical cell 
of the same volume because a flat shape allows for 

(A) a thinner cell membrane 

(B) greater cell mobility 

(C) greater plasticity of the protoplasm 

(D) greater structural reinforcement 

(E) a relatively greater surface area (Key E) 
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When the problem presented is clearly delimited by the wording of the 
test item so that the student selects not a universal solution but 
the best of the solutions offered 



Example : Of the following, which is an example of a simple machine 

designed to multiply distance rather than force? 



(A) a pair of pliers 

(B) a single movable pulley 

(C) a wracking bar 

(D) a cane fishing pole 

(E) a nutcracker 



(Key D) 



3* When the problem is such that the student is required to evaluate the 
relevance of five plausible, or even scientifically accurate, options 
and to select the one most pertinent. 



Example; Which of the following statements would be most helpful in 
explaining how a lens is able to bend light rays? 

(A) light reflects from the surface of glass 

(B) light travels more slowly in glass than in air 

(C) light striking a glass surface at a certain angle is 

reflected at the same angle 

(D) light is a form of energy 

(E) light can move through a vacuum (Key B) 
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4, When the problem has several pertinent solutions and the student 
is required to select the one inappropriate solution which is 
presented 

Example: All of the following are factors known to upset the law of 

genetic equilibrium EXCEPT 

(A) population size 

(B) natural selection 

(C) random reproduction 

(D) genetic drift 

(E) migration 

(Key c) 

General Suggestions For Preparing Five-Response Multiple-Choice Items 
1 - The test item in general 

a. Each item should test a concept or idea that is important for 
the student to know or understand. 

b. Items should vary in difficulty. For most tests, the difficulty 
of the "average item" should be such that 50-60% of the students 
get it right. In addition, there should be items which can be 
answered by as few as 15%, or as many as 857». Most writers of 
test items tend to underestimate the difficulty of the items they 
produce , 

c. The language of the item should be appropriate to the educational 
level of the students who are to answer it. 

d. The wording of the item should be simple, direct, and free of am- 
biguity. The wording should be edited for brevity. Unnecessary 
words are time consuming. 

e. Items containing double negatives are likely to cause confusion. 

If a word such as "noc" or ’’false** appears in the stem, try to 
avoid using another negative word either in the stem or in any of 
the options. 



2. The stem 



The stem of the item should present clearly the central problem 
or idea. The function of the stem is to set the stage for the op- 
tions which follow. The poor example which follows shows that if 
this principle is not heeded, the choices tend to become a group of 
not very closely related true-false questions. 

Poor: An ion 



(A) 

(B) 

(C) 

(D) 

(E) 



is any charges particle 

is formed when any substance dissolves in water 
is formed when atoms gain or lose electrons 
is present only in solutions which are being electrolyzed 
remits cnly when atoms gain electrons 
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(Key C) 



Good:_ Ions of calcium differ from atoms of calcium in all of the 
following ways EXCEPT 

(A) their ease of reduction 

(B) th*»ir ease of oxidation 

(C) their reactivity with water 

(D) their nuclear charge 

(E) the number of extranuclear electrons (Key D 1 ) 



3 • The options (possible responses) 

a. The options in an item are as important as the statement of the 
problem in the stem. They should be formulated with care; incor- 
rectness should not be the only criterion. The following types 
of incorrect options should be avoided: 

1. a response which is related to the situation and which sounds 
plausible to the untutored, but which is incorrect, 

2. a common misconception, 

3. a statement which, in itself, is true but does not satisfy 
the requirements of the problenij and 

a statement which is either too broad or too narrow for the 
requirements of the problem. 



The options should not include "specific determiners". Candi- 
dates who are "test-wise" can usually detect that statements con- 
taining words like "all", "always", "no", "never", and similar 
terms are generalizations so broad that they are likely to be 
false. Statements, on the other hand, which contain limiting 
words like "usually" or "sometimes" are more likely to be true. 

In writing options, one should try to avoid the use of "specific 
determiners" like these since some candidates, by logic alone, 

8 et credit for knowledge they do not possess. 



c. One must avoid t.-ording the correct response so much more care- 
fully than the other options that it becomes conspicuously longer. 
In general, the correct response should be no longer than the 
other options. 



XIV. GENERAL CHECKLIST FOR MULTIPLE- CHOICE ITEMS 

The following checklist may be used to check on whether the suggested 
procedures for test item development have been observed: 

1. Is the item stated as clearly as possible? 

2. Is the item, as a whtple, realistic and practical? 

ioa 
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Does the item, as a whole, deal with an important and useful aspect 
of the subject matter? 

4. Is each item independent of every other item in the test? 

5. Is the item, as a whole, specific? 

6. Is the item of appropriate difficulty? 

7. Does the item havg^a clear central problem? 

8. Is the item measuring something more than pure memory? 

9. Is the item stated accurately? 

10. Have the problem and alternatives been stated briefly, but 
completely? 

11. Does the problem contain only material relevant to the solution? 

12. Does the problem, or the statement, of the stem contain all the 
qualifying material necessary to provide a reasonable basis for 
selecting the correct response? 

13. Are the distractors (wrong responses) important, plausible answers 
rather than obviously incorrect distractors? 

141. Are all of the distractors appropriate to the item stem? 

15. Does the item stem avoid giving away the best response by irrelevant 
c lues? 

16. Do the alternatives overlap or include each other? 

17. If "None of the above", or "All of the above", have been used as 
alternatives, have they been used appropriately? 

18. Does any item contain "Any of the above" as an alternative? 



XV. LENGTH OF WRITTEN TESTS 

To test for all areas of knowledge or skill would require an extreme- 
ly long test. A well-designed test contains a sufficient number of test 
items to show the degree to which tradesmen and technicians possess the 
knowledge and skills in comparison with other individuals competent in 
the occupation. In test construction, the same principles are followed 
as in other areas of measurement; namely, that of sampling. The larger 
the number of samples, or items, selected from the various areas of trade 
knowledge and information, the greater the coverage of the whole content. 
At the same time, small areas in which a tradesman may not be thoroughly 
knowledgeable are reduced and will not unduly penalize him in the final 
test result. When competitors clearly Understand why a test that ade- 
quately samples the major fields relating to a job is more reliable and 
more valid than one restricted in coverage, they prefer tests sufficiently 
long to yield a reasonable appraisal of their competence. 
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In practice, the length of a test should be determined by the number 
of items necessary to establish the competence of the individual and to 
be within physical endurance limits of the candidates. Present practices 
require on an average from 2% hours to 3% hours for the written part. 
Economic considerations, administrative problems, availability of facil- 
ities, and accessibiity on the part of the candidates make it desirable 
to have the written and performance test conducted on a single day. 

The number of test items which make up an adequate sampling varies 
with each occupation. Any decision as to the number of items are a 
matter of judgment by both subject content and test experts. Two exam- 
ples are included in the Appendix serve to demonstrate as to how to deter- 
mine the actual numbers. 



XVI. PERFORMANCE TEST CONSTRUCTION 
Skills Essential to Competency Evaluation 

After the job description, an enumeration of all the factors to be 
considered In planning and developing a test is in order. Ideally, the 
tasks chosen for a performance test should be those which will decide 
success or failure on the job. The test constructor must look carefully 
at the on-the-job requirements and must resist the temptation of select- 
ing tasks which are relatively simple to observe and score, but which are 
of lesser importance occupationally. In developing a performance test 
for a cluster of occupations, care must be taken to avoid duplication of 
tasks. This can be done effectively by referring to the job analysis and 
then tabulating the essential skills which every tradesman or technician 
should be able to demonstrate. The work job should then be designed to 
test for essential work skills. 



Process-Centered Work Sample 

For occupations dealing primarily with procedures (such as the elec- 
tronic industries-~communication maintenance and repair), locating a 
fault and then correcting it represents the major tasks. The most effi- 
cient procedure to locate a difficulty is one that is systematic and 
logical. A candidate, not thoroughly familiar with all the steps in- 
volved in troubleshooting, might eventually find the trouble by chance 
or by some circuitous or illogical method. The use of systematic, logical 
and sensible methods supports the assumption that the candidate will find 
troubles in a variety of equipment. In this situation, it is useful and 
will Increase the discrimination and consequent validity of r-i* test if 
the procedure employed, as well as the method by which the c . .ilt is cor - 
rected, are evaluated. 
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Example of a Process-Centered Work Sample: 



Figure 2 represents a schematic diagram of an AC-DC five tube receiver. 
This is a good example of the equipment suitable for a more elementary perfor- 
mance test because it contains concepts and components which are found in on- 
the-job field operat.ions. While there are, of course, many more complex re- 
ceivers, the AC-DC receiver is representative to troubleshooting in more com- 
plex equipment. It is offered here only as an example of how a task in 
troubleshooting might be scored and recorded because it fulfills the majority 
of tt»£ occupational task requirements. 

The schematic diagram, voltage and resistance chart and block diagram are 
information available to a repairman on the job. 



Figure 2. SCHEMATIC AC-DC RECEIVER 
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One of the first s' < _>s in developing a troubleshooting test is to deter-? 
i m ^ ne where to start. If the rating includes starting the equipment, checking 
the fuse, visual inspection of the set each step offers little or no infor- 
mation for test purpose as to a candidate's proficiency. It was decided, by 
the test constructor, to have the candidate make only those troubleshooting 
tests that involved the use of the ohmmeter, voltmeter, and signal generator. 

The scoring sheet (Figure was developed after several formal tryouts 
and is the result of comparing several methods to determine which was practi- 
cal and discriminating. 



Figure 3, SCORING SHEET 
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A precise and straightforward check- by-check procedure of relevant 
troubleshooting steps was derived for each of five troubles. These 
steps were agreed upon by several experts, each of whom troubleshot the 
radxo receiver independently. Several additional checks (which although 
not basic were considered to offer some information) were also included. 

Each relevant check taken by the candidate was given 20 points- each 
associated but less relevant check was given 5 points. Time taken to 

troubleshoot and success in identifying trouble were found to be less 
discriminating. 



It must be stressed that this test and its scoring 
developed to meet specific needs. The scoring practices 
quate for other testing needs under different conditions 



procedure were 
may not be ade- 



Every occupation requires its own custom-built tests. The actual 
assignment for this work- sample procedure testing job was as follows: 

This receiver is properly aligned and in good condition with 
the exception of a single faulty component. This component must 
be identified and the trouble corrected. The proctor should be 
advised when the trouble is located. The evaluation will be 
based on the method used, the logic employed, the naming of the 
trouble after a few rudimentary tests, and time required 
(Time: minutes) 



C. Product-Centered Work Sample 

For some occupations like the construction industries or machine 
tra es, the specifications of the job clusters include the application 
A processes to the making of a finished part or product. 
Again, the job analysis is emphasized as a technique for establishing 
chose work skills and areas of information that are critical to the com- 
pletion of a work-piece or unit which becomes an important measurement 
or a craftsman s occupational competency. 

A detailed analysis for specifying a number of product-centered 

wor samples is illustrated and used in Chapter V for the machine indus- 
tries occupations. 

XVII. TIME LIMITS FOR WRITTEN AND PERFORMANCE TESTS 

The nature of the occupation, availability of the necessary facili- 
ties, travel time and distance for candidates to reach the testing center 
are ..ndirect factors which must be considered. Excessive time required 
may result in fatigue and adversely affect th, test results. Present 
practice in government, the Civil Service, even institutions of higher 
learning is to limit written examinations to three hours at one time 
In unusual cases, they may extend to four hours. 
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The j»erf ormance examination can rarely be conducted in less than four 
hours. More common practice requires between four to seven hours. It may be 
well to conduct the written and performance tests on two different dates. 

When a group is large enough, it may be divided with one group engaged in the 
written examination while the other carries on the performance. Aside from 
present practice, there are no specific guidelines for establishing time limits. 
The content and the skills of each occupation and the competency to be demon- 
strated determine the length of each toest. 
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Chapter V 



test administration 



I. 



general testing conditions 



tests™! perf!oraance Un can ? es f s of klnd «• administered, especially 

be free of o«“de dLtrLuo« be overemphasised. The test room should * 
illumination. ““"actions and comfortable as to temperature, air and 

familiar testing situation? g^^ll^t^nd?"??" 1 * 16 lnCroduced to an un- 

high speed. Just as newly hired work er? »?v^7^e7'° ^“tely at 

tion, it is sound practice to infrnlll hJ 8i J?" time to come U P «=° produc- 

setting. Simple test task, to b^ nf f ! " and L ida ^ gradually into a new 

the examiner, with no credit being give^for themV^^ 6 discussed b y 

«mg given tor them serve to ease the tension. 

There are good reasons for such practices. 

U) To allay anxiety; People react differently in stress si “u ati „ nc , 

• a candidate faced with the prospect of takine t 3 

termine his future, is under some tension. ^ assist i ll *7/ 
to relax and enable him to do his best a T assist a candidate 

able. 1 est ’ a warmup period is desir- 



( 2 ) 



rxL^^^r:uc\ re i% s ” c : ™ ere is 



and scope'of ‘th^examination^'whera^he is'ln^oort' '^ lcl ' **?'' 13111 the extsnt 
t1o 0 n 1 i?c e h d :„ n a e bS s '"hi i m' ^ »^L‘- 

have adequate time to p«p“e He should, also, 

Should be instructed as to what tn if 7 l examination. In addition, he 
Pected to bring tf th^er^^t^ and trials he is ex- 



both the writ^In^nneSorJfce 6 Cle * rly ” hat the candidate may expect in 

SSS, 2F - 



109 



o 

ERIC 



90 



II. EXAMPLE OF DIRECTIONS FOR MACHINE INDUSTRIES OCCUPATIONS 

(MACHINE TRADES) CANDIDATES 

A . General Information 

other°crl«rfr a J pro£1 = le " c J' examinations are used, in conjunction v.-th 
other criteria, for a single or for a combination of purposes; 

1. Admission to trade and industrial/technical teacher education 
programs, 

2. To meet state requirements for certification, and/or 

3. To establish evidence of occupational competence for advanced 
s anding in undergraduate or graduate programs of study. 

date in e h^ S n 8 r^ • teSt ^ * 6Vel ° f SkiU and knowled S* the candi- 
date in his particular occupational field, as compared with other exper- 

j.enced craftsmen in the same occupation. 

Two types of examinations are administered: 

The WRITTEN examination, consisting of multiple-choice test items 
covers the technical and related information, knowledge and judgements’ 
a competent tradesman applies to his work, and 

The PERFORMANCE examination includes a variety of typical tasks a 
competent tradesman is able to perform. 

natJ h :J C r ° f each / xam i nation lists the areas included in the exami- 
nation and the proportion of tast items from each area. 

B * Scope of the Written Examination 

sSuld"Sr ati ° n ’ the ° ry ’ fSCtS and baSlC ? rinci P les applicant 

1. Candidates may take three hours to complete the written examination. 

2. Candidates will be notified as to the date, time and place of the 
examination, 

3. A #2 lead pencil must be provided by each candidate. 

Percent of the Examination 

( 12 %) 

a * Bench an d Assembly Work, Layout and Inspection 



1) Principles and procedures of layout work 

Interpretation of blueprints, drawings, sketches 
) Principles of inspection and precision measurement 
' Utni2ation precision tools and measuring instruments 
5) Specifications from drawings for proper sequence of opera- 
, . „ t ions- treatment of materials-determination of tolerances 

) and tool, assembly and bench operations and utilization of 
proper tools " ' ~ i 
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b. 



Machine Sawing, Filing and Multiple Parts Processing 



(3%) 



1) Principles of operations 

2) Special functions - filing - internal and external 

3) Cutting tools, blades - friction cutting 

4) Selection of blades, file chains, etc* 

5) Determination of feeds and speeds 

c. Drilling, Tapping, Lapping - (Machines and Attachments) (57 0 ) 

1) Sizes and capacities of machines and attachments 

2) Single spindle and multiple spindle production operations 

3) Principles of precision finishing - reaming - lapping 

4) Determination of tolerances and internal finishes 

5) Principles and methods of clamping work 

6) Determination of speeds and feeds 

7) Determination of sizes and kinds of drilling and tapping tools 

d) Electrical Discharge Machining (47>) 

1) Principles of operation of electronic machining 

2) Operation and adjustment of controls 

3) Determination of limits and tolerances from blueprints and 

specifications 

4) Accuracy, tooling, size 

e) Grinding and Precision Finishing - Processes and Machines { 11 %) 

1) Types and kinds of machines and equipment 

2) Principles of operations - form and crush grinding 

3) Principles of clamping methods 

4) Identification and selection of abrasive wheels 

5) Kinds and types of finishes 

6) Principles of tool and cutter grinding; surface grinding 

f) Materials Treatment, Identification and Testing (57,) 

1) Characteristics of materials commonly used in the trades 

(ferrous - nonferrous materials) 

2) Machining characteristics (how to identify) 

3) Identification of sizes, specifications and grades of materials 

4) Heat treatment methods (hardening, tempering, annealing, carbur- 

izing, surface and spot hardening) 

5) Special heat treating methods (induction-atmosphere control) 

6) Hardness testing methods and instruments 

7) Cooling methods - water, oil, air 
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g. Turning Processes and Machines 



(17%) 



j. 



k. 



O 
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1) Types and kinds of machines (engine lathes, turret 

lathes, automatic turning machines) 

2) Speeds and feeds (adjustments and calculations) 

3) Clamping devices (mechanical- pneumatic chucks, collets, 

and face plates) 

4) Calculations for tapers, angles, thread turning 

5) Accessories - taper attachments - steady and follower rest 

6) Tools and instruments for single or multiple operation 

7) Methods of internal and external turning - single edge - 

box tools 

8) Production of single or multiple pieces 

9) Principles of turret and automatic turning 

Milling Processes and Machines (17%) 

1) Types, kinds and capacities of machines 

2) Principles of operation (conventional and climb milling) 

3) Accessories and special equipment (vertical heads, dividing- 

heads, turntables - application and principles of operation) 

4) Principles and methods of clamping regular and odd shapes 

5) Types, sizes and election of milling tools 

6) Calculation of feeds, speeds for various materials 

7) Principles of mechanical or optical contour milling 

8) Determinatioi. c>f milling finishes from blueprints and 

specifications 



Electrically-Controlled Machine Processes (N/C) (5%) 

1) Principle of operation 

2) Accessories 

3) Steps and procedures in planning the tape 

4) Sequences in tool setup 

5) Cutting feeds and speeds 

Shaping and Planing Processes (5%) 

1) Principles of operation - shaper - planer 

2) Application - advantages and disadvantages 

3) Methods of clamping regular and irregular shapes 

4) Tools and accessories needed 



Trade Computations 

1) The use of reference tables, handbooks and charts 

for calculations 

2) The use of formulas characteristic of the occupation 

3) Calculations applicable) to the work of the occupation 
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Trade Applications of Scienc e 



(5%) 



1) The theory and underlying principles of mechanical 

advantage 

2) Effect of heat and friction 

3) Elements of electricity and magnetism 

4) Pneumatic and hydraulic power 

5) Elements of chemistry, commonly applied to the occupation 

6) Other common applications of physical sciences 



m. Industrial Illustrations and Drawing Communication (5%) 

1) Basic principles of mechanical drawing, shop sketching, 

and technical illustrating 

2) Interpretation of complex drawings, layouts, plans and 

specifications 



C . Performance Examination - General Information 

The performance part of the examination provides an opportunity for the 
candidate to demonstrate his level of proficiency in the manipulative 
skills and judgements essential to carry on the work of the occupation. 

The scope of the performance part of the examination lists the skills, 
methods and procedures typical of the occupations from which the work 
sample jobs have been developed. 

1. Candidates may take five hours to complete the work jobs. 

2. Safety glasses are required and will be provided. Candidates may 
bring their own glasses. 

3. Personal hand tools, such as micrometers and steel rules may be used. 

4. Handbooks or reference materials may not be brought into the shop. 
They will be provided when necessary. 

5. Appropriate work clothes should be worn. 

6. Adherence to safety standards is mandatory. 

D. Scope of the Performance Examination 

Trade skills, methods and procedures of operation the applicant should 
be able to perform: 

1 . Bench and Assembly, Layout and Inspection Work ( 1 5%) 



O 
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a) Layout of work, including center, reference, contour 

and dimension lines, surface preparation 

b) Using common hand tools and measuring instruments, surface 

plate, and holding or clamping devices, precision tools 
a„d gages ^ 



C) Fi and S iapping nS * reamitlg ’ thread *ng, fitting, polishing 

d) Testing and inspecting with precision inspection tools - 
precision blocks, gages, indicators, hardness testers 
and comparator 



Mac hine Sawing, Filing and Multiple Parts Processing 

a) Set up and weld saw blades 

b) Saw to a line 

c) Set up and file to a line 

d) Clamp regular and irregular shapes 

e) Sawing and filing angular and irregular surfaces 

Drilling, Tapping, La p ping Machines and Attachments 

a) Set up work- vise, straps and bolts, jigs and/or a 

combination of accessories 

b) Drilling, reaming, counterboring, countersinking, 

tapping and lapping 

c) Grind drills 



Grinding and Precision Finishing 

a) Set up of work - magnetic chuck, vise, sine plate, 

magnetic accessories and other holding devices 

b) Selecting, truing and dressing abrasive wheels 

c) Surface grinding 

d) Slot or shoulder grinding 

e) Form grinding 

f) Plain external and internal grinding, taper grinding 

g) Cutter grinding - straight and spiral 

h) Selection of coolants 



Turning Processes and Machines 



a) 

b) 

'O 

d) 

e) 

f) 

g) 

h) 



Set up and operate engine and bench lathes 
Mounting of work, chucks, between center, collets, 
face plates ’ 

All basic external and internal turning operations 
Taper turning, all methods 

Drilling, reaming, tapping, boring, recessing 
Turning long and thin stock using follower and 
steady rest 

Multiple turning operations 
Sharpening all cutting tools 
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( 2 %) 



( 8 %) 



( 20 %) 



( 20 %) 



6. .Milling Processes and Machines 

a) Set up and adjust proper feeds and speeds 
k) Set up and operate the dividing head and rotary table; 
simple and compound indexing 

c) Set up vise and other holding devices for holding 

regular and irregular shapes 

d) Set up for various milling operations: slotting, 

angle milling, spiral milling, graduating 

e ) Set up for climb milling; 

7 • Electronic Control Operations - Optional 

a ) Set up tools in proper sequence 

b) Adjust machine for feed and speed 

c) Make a tape through numerical operation 

d) Mount tape and make a trial run 



8 . Planing and Shaping 

a) Set up work, tools, and machine accessories for 

plain surfaces, angular surfaces, contour shapes, 
slotting and serrating 

9. Material Treatment and Testing 

a) Hardening, tempering 

b) Annealing, normalizing 

c) Packhardening — Carburizing 

d) Testing for hardness 



HI* EXAMPLE OF DIRECTIONS FOR ELECTRONIC INDUSTRIES 
OCCUPATIONS- COMMUNICATIONS 



A. General Information 



Occupational proficiency examinations are used, in conjunction with 
criteria, for a single or for a combination of purposes: 

1. Admission to trade and industrial/industrial/technical teacher 
education programs, 

2. To meet state requirements for certification, and/or 

3. To establish evidence of occupational competence for advanced 
standing in graduate or undergraduate programs of study. 

S - >. '• 
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( 20 %) 



(5%) 



(5%) 



(5%) 



other 
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They are designed to test the level of skill 
date in his particular occupational field, as 
ienced people in the same occupation. 



and knowledge of the c a r* di- 
compared with other a^per- 



Two types of examinations are administered: 



The WRITTEN examination, consisting of multiple- choice 
covers the technical and related information, knowledge and 
a competent tradesman applies to his work, and 



test items, 
judgern^ts 



The PERFORMANCE examination includes a variety of typical task.? a 
competent tradesman is able to perform. 

The Scope of each examination lists the areas included in thr exami- 
nation and the proportion of test items from each area. 



B. Scope of the Written Examination 

1. Candidates may take three hours to complete the written examination. 

2. Candidates will be notified as to the date, time and place of the 

examination. * 

3. A #2 lead pencil must be provided by each candidate. 

1# Basic Electrici ty as Applied to the Electronic Industry 

a ‘ D.C. Circuits (107) 

1) Series 

Characteristics and calculations 
(voltage drop, total current, power, total 
resistance, etc.) 

2) Parallel 

Characteristics and calculations 

3) Combinations 

a) Series - parallel 

(equivalent resistance, voltage drop, total 
current, power, total resistance, etc.) 

b) Voltage divider calculations 
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b 



(5%) 



# Measurements, Standards and Tolerances as Applied to 
Voltmeters, Amme ters, Ohmmeters, and Oscilloscopes 

1) Calculation of shunts for ammeters 

2) Calculation of multiplier resistors for voltmeters 

3) Meter sensitivity and loading effects 

4) Ohmmeter circuitry 

5) Oscilloscopes as voltmeters 

6) Determination of unknown fre H ^ncies with an oscilloscope 

7) Observing waveshapes with an oscilloscope 



c . Conductors, Insulators, and Semi-Conductors used in 
Electronic Devices 

1) Factors affecting resistance of copper conductors 

2) Factors affecting conductivity of semi-conductor 

3) Effectiveness of various materials as insulators 



d . Inductive Devices used in Electronics 

1) Transformer operation and calculations 

(phase, voltage, and current relationships, regulation) 

2) Radio- frequency inductors 

(rf , if, powdered iron and air core, single and double- 
tuned) 



e . Alternating Current in Electronics 

1) Basic concepts 

(RMS, effective, peak, peak to peak values) 

2) Effect of inductance in an AC circuit 

3) Effect of capacitance in an AC circuit 



f . Single Phase Circuits 

1) Series - R,L.C # 

2) Parallel - R,L,C 

3) Resonance - Series, Parallel 

4) R-L and R- C time constants 



g . Batteries as a Source of Voltage for Electronic Devices 

1) Primary cells (zinc-carbon, alkaline) 

2) Secondary cells (lead-acid, nickel- cadmium) 





( 2 %) 



(47o) 



(5%) 



(137o) 



( 17 .) 
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Electronic C 



Applied to Vac uum Tube and Solid State Device s 



a , Basic Aspects 

1) Static characteristics 

2) Dynamic characteristics 

3) Biasing 

4) Translator characteristics 

5) Other semi-conductor devices 



(17%) 



b * Electronic Circuits 

1) Power supplies 

2) Vacuum tube amplifiers 

3) Semi-conductor amplifiers 

4) Oscillators (including multivibrators) 



(31%) 



c ' Amplifiers in Cascade for RF an d AF 

1) Coupling methods 

2) Decoupling 

3) Frequency response 

4) Inverse feedback 

5) Gain or attenuation (db) 

6) Voltage vs. power amplifiers 

7) Optimum transfer of energy 



d ' Other Con cepts in Electronic Application 

1) Amplitude modulation 

2) Frequency modulation 

3) Pulse definitions 

4) Logic concepts 



3. 



Ba sic Methods and Pr o cedures Applied to Mainten ance 
C orrection and Adjustme nt of Units. Component Parts 



j. 



a . 


Trouble 


identification 


b. 


Methods 


of correction 


c. 


Methods 


of maintenance and adjustment 



(25%) 
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culation*% m w- USe J alide rule as an aid in performing mathematical cal- 
u. a^ 1 i \" 8 written or Performance examinations. The Allied Data 
Handbook may be consulted during -he performance examiner 
no other notes, reference books or other printed matter ^ie used 

IIS 
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C . Performance Examination 

1 . General Information 

The performance part of the competency examination provides an 
opportunity for the candidate to demonstrate his level of proficiency 
in the manipulative skills and judgements essential to carry on the 
work of the occupation. 

The Scope of the performance pari- lists the typical maintenance 
assignments encountered in radio, television, recording and other 
communications equipment. 

a. Candidates may take five hours to complete the jobs. 

b. The Allied Data Handbook may be used during Che performance 
examination. 

c # Personal tools may be used. 

d. Appropriate work clothes should be worn. 

e. Adherence to safety standards is mandatory. 

f„ Speed of diagnosis and efficiency in repair techniques will be 
part of the candidate’s performance evaluation. 

2 . C cope of the Performance Examination 

Candidates must complete six jobs in the areas as listed. Both 
vacuum tube «*nd solid state equipment will be presented for repair or 
examination. Problems, which normally )ccur in service, will be used 
to test the candidate’s ability to make diagnoses and repairs gener- 
ally employed to correct the faulty circuit. 

a . Radio Equipment Perform two of three jobs 



1) Component failures 

2) Adjustment of service controls 

3) Sound section repairs and adjustments 

4) Sweep circuit repairs and adjustments 

5) R-F oscillator tuner adjustments 

6) Color receiver field adjustments 

7) Trap adjustments (4.5 mil) 



1) Alignment of AM and FM radios 

2) Component failures (shorts or opens) 

3) Repair of low-power transmitters 



b # Television Service 



Perform three of four 

j ° bs 
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Recording Equipment 



c . 



Perform one of two jobs 



1) Adjust mechanical malfunctions of record 

changers or tape transports 

2) Knowledge of cleaning or drive mechanisms 

of record changers and tape; transports 



Performance of the job tasks will require the use of common hand tools of 
the trade, including proper alignment tools. The candidate should bring the 
hand tools to the examination. 

The candidate must be familiar with the use of the necessary test equip- 
ment to perform the assigned work, such as: the VTVM, VOM, oscilloscope, 
various radio and audio frequency generators, signal tracers, and dummy loads. 
Problems involving tube failures will not be included in any one of the test* 
items. The knowledge of the use of spec 1 ' * equipment for color television 
setup may be required. Equipment will be wished. 



IV. INSTRUCTIONS AND DIRECTIONS FOR EXAMINERS 

A. Examiners should be furnished with a list of names and Social Security 
Numbers of candidates and the specific tests to be given. 

B. Candidates should identify themselves by their letter of notification. 

C. A candidate who fails to present his identification may be admitted to 
the examination, if his name is on the list. 

D. Candidates who appear more than 45 minutes late may not be admitted. 

CJnder exceptional conditions, admission to the examination may be granted 
and the admission time extended. 



A. Directions for Administering Written Examinations 

1 . Things for the Proctor to do Prio r to the Examination 

a. Obtain specific directions ’^lative to proctoring the written 
examination. 

b. Obtain list oj candidates expected for the examination. 

c. Obtain er unination papers, answer papers, and other material 
needed for the test. 

2 . Before the Examination Begins 

n • Check conditio^ of the examir.a^ ! room for proper heat, light, 
ventilation am ^uate seat! ^ ncilities, and enter any 
irregularities Ou the Fxrnvrat .Import. 

* 

Require candidates to identify themselves as they arrive. 
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b. 



c. Seat candidates taking the same examination at a distance from 
one another. 

d. Candidate may bring only those supplies specifically required 
for the examination. 

e * Check to be sure each candidate has a pencil. 

f. If other supplies, such as Handbooks or slide rules are 

suggested, the proctor should have a reserve supply available. 



Distribution of Examinations and Oral Instructi ons 

a. Be sure all candidates are seated and ready to begin work. 

b. Make any special announcements that may be required. 

c. Hand to each candidate, the appropriate examination envelope, 
requiring the envelopes be kept closed until the "start" signal 
has been given . 

d. Say to the candidates, "Take the examination out of the envelope. 
(Pause) Put your Social Security Number on the line for the Name 

or Number and today's date, which is . Do not put 

your name on the booklet, unless you do not know your Social 
Security Number. (Pause) Make sure your Social Security Number 
is entered on all work you hand in to the examiner. 

After you read the directions, begin work. You have three (3) 
hours during which to complete the examination. Mark your answers 
carefully in the spaces provided on the answer sheet. If you make 
an error, erase your mark completely. Failure to do so, may c 
suit in an incorrect answer being recorded. Also, note th.it the 
answers are in letter form and are arranged from left to ri;*ht (or 
in whatever form they may be arranged). After completing the £-<•- 
amination and are ready to check out, place all your work and 
papers into the examination envelope. You may not leave the room 
for any reason, without first checking with me (or the proctor)." 



What to do During the Examination Period 

a. Note the time when the examination begins. 

b. Only one candidate may be permitted to leave the room at any one 

tie. 



c. No candidate should leave the room, after anoti or person taking 
the same test has chocked out. 



d. 



Be courteous if questions arise, but do not interpret the intent, 
or meaning, of a test question, or furnish any information to the 
candidate. Suggest that the candidate answer the test item as 
best he can. 
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e. If questions arise concerning possible errors in items (e.g. 
spelling, wording, ambiguity or errors in the alternatives), do 
-P- Q . t correct the error or announce it to the group. Again, 
tell the candidate only, 51 1 cannot give you any information. 

Answer the item as best you can." (Do this even if there is no 
item error.) However, enter any possible error on the Examina- 
tion Report in the Irregularities Section. 

f. Be a? er t and circulate quietly around the room, continuously re- 
maining in the room to supervise the examination. 

g. Stop work at the expiration of the allotted time, except for those 
who have official permission to continue. 

h. As the examinees chsck out of the examination room, be sura that 

(a) the Social Security Number is on the examination booklet 
and all other papers, and 

(b) all test ma terials are put Into the candidate* s envelope. 

i. Return all examination papers to the appropriate authority. 



Directions for Administering Performance Examinatio ns 

Performance examinations require far greater preparation than 
written examinations. It 1 ^ absolutely essential that the following 
steps be carefully followed. Deviations from these procedures will 
result in difficulties in administering and scoring the examination. 

As you complete each of the following steps, check the item off cn 
the Examination Report form (page 109) and, also, record any problems 
or irregularities. In this way, you will be adequately prepared for 
the examination, and will avoid needless hardships and t ime- consuming 
delays. 

If you encounter any difficulties which you cannot handle in pre- 
paring for the examination, contact the appropriate person immediate ly , 

so that the problem can be solved nrior to the test date. 

will check, prior to the date of the examination, to 

be sure that everything is ready and will, also, check with you dur- 
ing the actual i dininistrat o.on of the performance examination since 
unexpected problems may arise. 



1. Facilities for Examinations 



Performance tests should be conducted in shops or laborator- 
ies equipped with appropriate and sufficient machines, equip- 
ment, hand tools and materials. Shops may be located in voca- 
tional schools or selected industrial establishments. 

Specific directions for conducting examinations for each occu- 
pation v/ill be placed in the hands of the examiner well in 
advar.ee of the date of tho examination. 
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b. 



In any case, the examiner should analyze the test problems to 
determine amount and type of tools, equipment and other items 
needed for the number of persons scheduled to take the test. 
When the best available facilities are inadequate, arrangements 
should be made to supplement those at hand by securing addi- 
tional equipment in order to conduct the test under the best 
possible conditions. 



2. Supplies for Testing 

to the examina t ion the examiner should determine the 
materials and supplies needed lo conduct the tests. In cases where 
supplies are not readily available and must be purchased, the exam- 
iner should consult 



3 - What to do Before the Day of the Examination 

a * Read the "Directions to the Examiner" and "Performance Exami- 
nation" for the examination being given. 

b. Check to verify arrangements for time and pJ-\ce of the 
examination. 

c. Visit the testing center to see that all equipment, tools, etc., 
necessary for the test is available. 

d. Check all equ pment to make sure that it is in good working 
condition. 

e. Set up all tools, machines, equipment and working conditions 
so each working station is as nearly identical as possible. 

f. Prepare all partially completed test pieces and/or defective 
equipment for troubleshooting for the examination. 

g. Check to make sure that all tools, materials and supplies are 
in a convenient location for each examinee. 

h. Place all test materials (booklets, answer sheets, scrap paper, 
rating forms) in a large envelope for each candidate, an d mark 
it with »-he candidate 1 s name rnd Social ** e cur i ty Number. 

i. Familiarise yourself with the scoring procedure. 



THESE STEPS Ai;E TO BE COMPLETED AND RECORDED ON THE "EXAMINATION REPORT' 
PRELIMINARY CHECK-LIST FOR PERFORMANCE EXAMINATIONS: BEFORE THE DAY OF Thf 
EXAMINATION . 1 1 (See sample form page 107) 
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/+ • What to do During the Examinat ion 



a. Before actual work begins, permit the examinee to acquaint 

himself with his surroundings since the shop may be strange. 
Point out the location of such facilities as: electrical out- 

lets, service facilities, and other physical details. 

b. Note the time each job is started and finished by each candi- 
date . 

c. Observe the candidate at his work. Check his work methods 
quietly and inconspicuously. Keep a record for your own pur- 
poses as the examination progresses. 

d. Spend approximately the same amount of time with . ach candi- 
date . 

e. Check and rate ch step of the job as it is completed by 
each candidate. 

f# Tag or stamp each completed job with the candidate ’ s examina- 
tion number. 

g. Maintain an impartial attitude at all times. It eliminates 
criticism and aids in reducing tension which, quite naturally, 
develops during the examination. 

h. Provide Handbooks, reference tables, etc. requited in the ex- 
amination. 

i. Carefully record all ratings required on the "Performance 
Rating and Summary" form for the specific field. If there 
is no special form, use the General Performance Rating and 
Summary form. (See Figure 4, page 107.) 



5. What to Avoid During the Exam ination 

a. Allow only one examinee to leave the room at any time. 

b. No work is to be taken from the shop during the examination. 

c. Avoid interpreting the meaning or intent of a job assignment. 
Suggest that the candidate proceed according to his under- 
standing. 

d. Rating sheets jhould be marked only when out of the candi- 
date's presence. 

e. Refrain from giving any information to a candidate concerning 
his progress or the probabie result of the examination. 

f. Such matters as ^a ^candidate's occupational 
education, or perSxiriial pi'oblems should not 
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experience, his 
be discussed. 



At the Close of the Examination 



a. Stop all work at the expiration the maximum allotted time. 

b. Collect all examination forms and directions. 

c. Identify all test pieces with the respective candidate's 
number . 

d. Retain all finished work. 



Completing the Scoring of the Examination 

a. Most of the performance ratings can be made during the examin- 
ation. 

b. Inspect and rate all other completed work immediately fter 
candidates har departed from the shop. 

c. Use only the performance rating sheets provided. 

d. Enter comments which have a bearing on tne competency of the 
candidate on the back of the sheet. This is particularly im- 
portant in cases wL e the performance is below average. 

e. Sign the rating sheet in place provided for 11 Examiner 11 . 

f» Return all scoring sheets, question papers, pri :s and other 
materials provided for each examination. 

g. All totals will be computed at the National Occupational Com- 
petency Testing Project Center. 
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SAMPLE EXAMINATION REPORT 



Figure 4. 

PRELIMINARY CHECK- LIST FOR PERFORMANCE EXAMINATION 
Complete each step as listed, filling in the date when complete^. 



2 . 



b . 
4. 



5. 

6 . 



7. 



8 . 



9. 



Have read "Directions to the Examiner 11 which are specific 
to the examination being given. 

Have read the " Performance Examination", which is the 
actual examination the candidates take. 

Arrangements for the time and place of the examination 
have been verified with the Regional Coordinator. 

Visited testing center to be sure that all equipment; 
materials, and tools necessary for the test are available. 
(See: "Directions to the Examiner" and the "Performance 
Examination" for the specific items that will be needed.) 

All equipment has been checked and is in good working con- 
dition. 

Tools, machines, equipment and working conditions have 
been set up, as nearly identical as possible, at each work 
station . 

All partially completed test pieces and/or defective equip- 
ment for troubleshooting are prepared and ready for the 
examination. 

Tools, materials. Handbooks and supplies are in a conve- 
nient location for each examinee. (This should be similar 
to the typical occupational work situation where he would 
select equipment from a choice of tools and supplies nor- 
mally available. However, the examinee should not be re- 
quired to spend time in securing stock and performing 
other preliminary work.) 

Job assignments, necessary drawings and other written di- 
rections, relating L the test, have been placed in an 
envelope and marked with the candidate 1 s name and Social 
Security Number. Partially completed items are available 
for the candidate. 



I have completed the above steps, completely familiarized myself with all pro- 
cedures I am to follow during the examination and have verified this with the 
Regional Coordinator. 




Signature 

(Date) 

(If there are any problems relating to these steps, or if you see any problems 
in administering the examination, contact the coordinator of the testing pro- 
gram.) 
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Chapter VI 



WRITTEN TEST RATINGS AND PERFORMANCE EVALUATION PROCEDURES 



The work of the tradesman involves the use of. instruments for precision 
measurement. Tests pre, likewise, measuring devices to provide accurate inr 
formation regarding an Individuals occupational competence. To achieve ac- 
curacy, the manner in which test results are recorded requires careful treat- 
ment. 



I. RECORDING TEST ANSWERS - WRITTEN TESTS 

Modern data processing equipment makes It possible to record, tabulate, 
and rate test results with remarkable speed. Processing with this equipment, 
also, setc a pattern fox' recording the answers and solutions to test items of 
written examinations. There is no one precise form for all examinations be- 
cause of the many diverse methods of recording which are used. Whatever meth- 
od Is employed, precise directions must be given to the examiner on how to 
proceed. For the development of a recording form, the following must be 
taken Into consideration: 

A. For Machine Cards 

1. What are the limitations of the equipment? 

2. How many spaces are available on one card when scanning equipment is 

used? 

3. What physical arrangement Is required for the layout of the card? 

B . How Should Test Results Be Reported? 

1. What kind of computer facility is available? 

2. What kind of program must be planned to provide: 

a) Listing of candidate's data (desired background) 

b) Grades In numerical or letter form 

c) Statistical treatment of data: 

1) Item analysis 

2) Difficulty level and discrimination analysis 

3) Evaluation of various parts of the examination 

4) Determination of correlation coefficients 

3) Determination of control tendencies and standard deviation 

6) Other Information desired about the test or the relative 
standing of the individual 

Data processing equipment and facilities are continuously improved. Before 
a method of recording Is established, it would oe wise to examine existing 
procedures In the field for the latest equipment available. Recent develop- 
ments permit the use of larger answer sheets - 8^ x 11" instead of the 
smaller IBM card. The guiding consideration should be to strive towards ob- 
jective rating by objective, mechanical means. 
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II. RECORDING TEST RESULTS - PERFORMANCE TESTS 



Evaluating the work, per form&uce and recording the results of a performance 
test is more difficult because subjective judgements must be recorded as part 
of the overall evaluation- Determining factors for developing a rating form 
are the nature of the trade, the kinds of skills and judgements to be evalu- 
ated, the relationship between objective test items and subjective observa- 
tions, the ease or difficulty in observing and rating the various aspects of 
the performance, etc. 

Figure 6 . TROUBLESHOOTING EVALUATION FORM 
Place Name __ 



Examiner Test Point 



Step-by-Step Sequence Check 



VI- R V SS 



1 














to 
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V2 
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2 














to 
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to 
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V5 














1 
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to 














8 














Total 















Date 

Component Check 

CAPACITORS 



Cl 



to 




CUB 


REGISTERS 


R1 


to 


; 


R9 


TUBES 


VI 


to 


: 


V5 


TRANSFORMERS 


COIL 



LI 



1) A precise check- by- check procedure of relevant troubleshooting steps 

were derived for each of five troubles (not necessarily including 8 items) . 

2) Each relevant step was assigned 20 points, 

3) Each associated step, although not basic was assigned 5 points. 

4) All irrelevant steps were assigned 5 points (to be subtracted from the 
total) . 
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Evaluation of Troubleshooting Items 



If troubleshooting represents the major portion of the test, a list- 
ing of all the steps involved should be developed, including a space for 
check off. Possible values for each step completed or not completed 
should be included. A separate rating scale may be set up for the per- 
formance of the connected unit. A step-by-step evaluation form follows. 

Evaluations Including Special Time Factors 

Should speed be a vital factor, then :he maximum time permitted 
should be carefully determined and penalties indicated for those candi- 
dates who exceed the time limits. For candidates completing the tasks in 
substantially less time premiums for early completion must be indicated. 

A rating form for evaluation of speed follows: 



Figure 7. TIME STANDARDS FOR JOB 



Place Date 

Examiner Name 



1 

Time/Minutes 


Rating 


25 


90 


30 


85 


35 


80 


40 


75 standard tine 


45 


70 


50 


65 


55 


SO 



TOTAL SCORE 



Evaluations Including Measurements of Accuracy 

For instance, if a part must be machined to a tolerance of + 0.001" 
on a dimension of, say, 7.000", the candidate who machines the part be- 
tween 6.999" and 7.001" may receive a perfect score. A candidate who ex- 
ceeds these maximum tolerances would score below a passing grade. Other 
ratings between 100 and 70 should be established in advance for measuring 
dimensional accuracy to identify scores to be assigned for gradation in 
accuracy. 

If the sama examination stated that one objective was to rate the 
candidate* s competence to machine to precision limits of one ten thou- 
sandth part of an inch (0.0001") and closeness to the base dimension was 
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important, then a different rating scale might be applied. The candidate 
who machined the part exactly 7.000" would receive a perfect score. The 
+.001" tolerance might be divided into ten increments representing a per- 
fect score of 100 and a passing score of 70. Thus, the candidate who 
machined the part to 6.9999" (-O.OOOx") or 7.0001" (+0.0001) would receive 
100-3 or 97; 6.9998 or 7.0002", 94, etc. 

Also, for those exceeding the limits of tolerance (+ 0.001") a series 
of scores could be established below the passing score to indicate limita- 
tions for accuracy of machining. Thus, the scores permit differentiation 
between those with occupational competence to work to the nearest + 0.0001" 
and others whose skill is limited to + 0.001". 

The same techniques may be applied to other trade or technical occu- 
pations which may not require the same degree of precision measurement. 
Directly measurable items have the advantage of objectivity and fairness. 
Such items as dimensions, location and sizes of cut-away sections, sizes 
of threads, etc. can be measured. The important point is that tolerances 
should be given and values assigned to represent a range of ratings which 
differentiate the capable occupationally competent craftsman from a less 
skilled worker. 

Figure 8. DIMENSION RATING FORM 



P lace 


Job 




Date 




Examiner 






Name 










Dimension 


Specimen Dimension 


Weight 


Rating 


r 

Remarks 


A 




5 






P 


_ ^Yv 


10 






C 


- 


10 




- 




D 


_ ... 


10 




<v* c 'ITv. ^ 

4 3-g y1°^ 


1 










1 










1 










K 




15 






Threads 




15 






Workmanship 




20 






Usability 




15 








1 








TOTAL 


100 
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Evaluations of Finished Product 



For finished product emphasis, careful distinction must be drawn be- 
tween methodology employed and the items of the finished part to be 
measured. Every effort must be made to define such terms as workmanship, 
work habits, work methods. Sometimes, a large number of factors are as- 
sumed under the heading ’’skill 11 . The latter may cover analysis, diag- 
nosis and planning the job, selection of appropriate materials, tools, in- 
struments, etc. Often, the category also includes correctness of pro- 
cedures, interpretation of drawings and other directions. The list could 
be extended considerably. 

Obviously, an evaluation expressed in a single number on a rating 
scale, or an identification as; unacceptable product, marginal quality, 
average , good, quality product/ would be extremely subjective and would 
inevitably lead to substantial differences among the examiners. 



E. Evaluating Proper Work Methods 

The completion of the product indicates certain occupational compe- 
tence. In addition (for tradesmen interested in teaching), work habits 
acquired over a period of time and the methods employed in producing the 
product are significant parts of overall competence. Scoring forms to 
evaluate procedures and work habits might be developed in several ways. 

1« A common type of rating form lists each 3tep required for a particu- 
lar job. The form illustrated in Figure 9 simply requires a "Yes" 
or "No” check that later is considered as indicating -whether or not 
the candidate follows correct work steps or procedures. 



Figure 9. RATING FORM FOR C KING SEQUENCE OF STEPS 



Trade ite 


Examiner 


Candidacy I dent if ication 




YES 


NO 


Step 1 







Step 2 






Step 3 






Step 4 






Step 5 






Step 6 




— 
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2. A different evaluation form may list a series of statements of the 
steps employed, followed by columns to record a number or letter 
grade to indicate how well performed and safety precautions followed. 
The examiner evaluates the performance against each statement and re- 
cords his evaluation. The overall judgements are then totalled. 

3. Still another form may be used to evaluate particular phases of a job. 
While a job may be completed within all tolerances, the finished pro- 
duct may not shov7 how many additional steps or operations had to be 
performed before the final cut was taken to get to the required dimen- 
sion. In such a case, each additional cut or operation is noted and 
points are subtracted for this extra work. 

4. Sometimes, a simple graphic scale, with values from poor to excellent, 
merely requires the examiner to note the term which best describes 
the candidate's performance on an operation cr series of work pro- 
cesses . 



Example : Clamping Work i Lece 



12 3 4 



5 



Level Unfamiliar 

of with 

Performance Process 



Unsure 

of 

Steps 



Average Capable: 

Performance Above 

Average 



Excellent 

Craftsman 

Techniques 



Such items as safe practices, orderly arrangement of tools and mater- 
ials, machine set ups, and many others can be arranged in similar 
fashion. 




Attempts have been made to design one rating form for all trade per- 
formance tests. However, desirable this may seem to be administra- 
tively, the differences among the various trades, their special charac- 
tistics, and the need for objectivity requires a special rating form 
for each occupation. The two forms developed for the National Occu- 
pational Competency Testing Projects for the Machine Industries Occu- 
pations (Machine Trades) and the Electronics Industries Occupations 
(Communications) appear is Figure 10 and Figure 11, respectively. Note 
that on Figure 11, all jobs are recorded on one rating form. The final 
score will be the sum total for all rated jobs completed divided by 
the number of jobs. A suggested mating distribution is provided on 
the back of the form. The examiner is also asked to express his 
judgement, based on observations, of the candidate's performance. 

F . Examiner Reliability 

Despite all efforts to eliminate subjective judgements of examiners 
in evaluating the performance of a candidate, certain job factors cannot 
be appraised in any other way except by personal judgement. Experience 
has shown that training examiners with proper focus on the items to be 
evaluated and rating scales can produce agreement among different exam- 
iners. A high factor of reliability among examiners evaluating perfor- 
mance tests has been found when these conditions are met. 
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Figure 11 



NATIONAL OCCUPATIONAL COMPETENCY TESTING PROJECT 
OCCUPATIONAL COMPETENCY TEST 
ELECTRONICS INDUSTRIES OCCUPATIONS (COMMUNICATION) 

EXAMINER'S PERFORMANCE EVALUATION SCALE 
Candidate's Name and Number 



P lace 



While the candidate is performing the assigned tasks, rate his performance on 
each job on the scale shown below. Indicate any weaknesses in the left column. 

The candidate should be rated on two features of his performance: a) his work 

methods, and b) the finished job. The following seven aspects of his work 
methods are to be used: 

1. Does he approach the identification and location of the trouble 

systematically in a step-by-step manner? 

2. Does he use the accepted trade method in correcting the trouble? 

3. Does he select test equipment properly? 

4. Does he perform hand-tool operations skillfully? 

5. Does he work neatly, accurately and safely? 

6. Does he work at an acceptable speed? 

7 . Does he avoid excessive and inappropriate use of reference 

mater ia 1? 

In assigning ratings to the job performance, remember that the scores indi- 
cated are maximum and should be given only to the extremely competent candi- 
date (for a breakdown see next page) . 



Work Methods and Finished Job 


Maximum 

Score 


Jobs 


Comments : 


1 


2 


3 


4 


5 


6 


7 


8 


Weaknesses 


a- 1 Procedure in identifying 
and locating trouble 


20 




















2 Method of correcting 
trouble 


15 




















3 Selection and use of 
test equipment 


10 




















4 Skill of hand tool 
operations 


10 




















5 Neatness, accuracy and 
safety of work 


5 




















6 Speed 


5 




















7 Use of reference materials 


5 




















SCORE FOR WORK METHODS 


70 




















b- SCORE FOR FINISHED JOB 


30 




















Total Score 


130 





















Final Score = Sum of the scores on all jobs 

Number ^of^J-obs 



Examiner 1 s Signature 
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These are estimated average values. A rater should go above or below these 
figures to more accurately record his judgement. 

Figure 12. BREAKDOWN OF FIGURE 11 



Level of Competence 


Rating According to 
Value of Maximum Score 




30 


20 


15 


10 


5 


Worker inept or out of practice 


6 


4 


3 


2 


1 


Worker with below average and 
limited facility 


12 


8 


6 


4 


2 


Average worker in the field 


18 


12 


9 


6 


3 


Good or above average worker 


24 


16 


12 


8 


4 


Extremely able, highly competent 
worker 


30 


20 


15 


10 


5 



State your overall impression of the candidate 1 s competence as a worker in 
the field, in addition to the numerical score. 
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Chapter VII 



EVALUATING OCCUPATIONAL COMPETENCY TESTS 



When a craftsman has to take measurements requiring close tolerances, he 
uses a precision measuring instrument, gage, etc.. Repeated measurements of 
the same dimension or unit produce the same result. Therefore, it is a re- 
liable instrument. Electr ical, mechanical , hydraulic, and other measuring de- 
vices function continuously with?n certain limits of accuracy and are reliable. 

On the other hand, measuring occupational competence involves many more 
variable factors, is far more complex, and the criteria for which competence 
mxict be measured objectively are much more difficult to establish. However, 
methods and techniques have been developed which assist the test constructor 
to check the quality of the measuring instrument. 

A test must measure not only what it is intended to measure, but it must 
also be valid for the purpose it is to serve. Occupational competency tests 
are intended to measure the proficiency level of an individual relative to the 
proficiency standards demanded in the occupation. 

In some states, the number of years of experience an individual has spent 
in his field and an evaluation of job performance by supervisors and employers 
is used as an indication of job competence. The subjective nature of this ap- 
proach has led tc a more positive direction for determining criteria against 
which occupa' onal competency is to be measured. Specific information for 
test development is identified from job studies, job descriptions and compre- 
hensive analytes of occupational job clusters. Occupational skills and es- 
sential informat ion which are characteristic of the proficiency at a certain 
level of performance have been identified and serve as the criteria of what a 
competency test must measure. 

This chapter deals with the evaluation of occupational competency tests 
in terms of validity, reliability, analysis of test scores, difficulty level 
and discrimination of test items, length of tests, item analysis, and test 
standards. 



I. KINDS OF VALIDITY 

Tests are used for a variety of purposes, each requiring a certain type 
of validating evidence. For purposes of this report, four types of validity 
will be discussed. These represent accepted practices by the American 
Psychological Association, the American Educational Research Association, and 
the National Council on Measurements Used in Education. 

A . Content Validity 

Content validity is most appropriate with achievement evaluation. 

The question to be answered is primarily, "Does the test r.easure the 
skills and technical information which are character istic of the competent 
craftsman?" . A test has content validity when it contains a representa- 
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tive sampling of the manipulative skills, the trade information, and 
"know-how" , at the journeyman level. Content validity relates to the ade- 
quacy of the test, as being representative of an area of behavior, such 
as; performing certain tasks within time and accuracy limits. Content 
validity is a matter of the occupational specialist who has the overall 
knowledge and skill of the occupation making a determination on content 
adequacy . 



B . Concurrent Validity 

Concurrent validity is a relevant concern in the evaluation of 
achievement. It represents a comparison of test based categories and 
actual achievement data in the occupation. Concurrent validity is, thus, 
expressed in terms of the relationship between test performance and an 
accepted contemporary criterion. For occupational competency tests, this 
represents a very difficult problem. At this time, no national perfor- 
mance criteria or standards are available. Instead , the computed corre- 
lation coefficient, based on test results, provides a quantitative 
measure of this relationship. 

C . Predictive Validity 

The predictive validity of an instrument, requires criterion against 
which pi'edictions can be compared. While much work has been done in the 
academic subject content area, and tests for supposedly predicting college 
success are widely used, occupational competency tests are not intended to 
measure future success as teachers. They serve only to establish an indi- 
vidual's proficiency in his occupation. 



D . Construct Validity 

Construct validity concerns the extent to which a test indicates the 
meaningful characteristics of the occupation. - Information about such 
characteristics provides an understanding of the tradesman's performance 
level on the job. The content validity of an occupational competency test 
is judged on the basis of how adequately the test samples (represents) the 
total content of the occupation. Construct validity, however, concerns 
the test’s ability to measure the individua 1 ' *> actual achievement in the 
occupation. A competency test of high construct validity must distinguish 
between a tradesman and others performing at different levels of capa- 
bility. 




The following kinds of evidence are useful in establishing the 
construct validity of a measuring instrument: 



1. Expert judgement concerning the extent to which responses to 
the test items provide knowledge about the individual's pos- 
session of the skill and information; 



Correlation with other tests, particulp* Ly 
accepted measures of the sam% -construct ; 
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2 . 



those that are 



3. Correlation with other characteristics of the individual; and 

4* Correlation with factors in the individuals environment which 
might affect test performance (types of employment, kinds of 
equipment different from test facilities, etc.). 



In essence, the focus of construct validity is upon the validity of 
the specifications derived from the occupational analysis. For instance, 
if occupational competency is to be measured for the person who is to 
teach in the machine trades, such characteristics as operating skills and 
trade knowledge related to the various machines, tools, apparatus, and 
materials, methods of work, etc. (which are characteristic of a highly 
competent individual), must be included in the test. 



II. VALIDATION OF TESTS 



Logical Approach 

In occupational competency testing, two questions are of major 
concern. The first, "Is the test suitable?'* and the second, "Will the 
test scores be reliable?" Discussing the various aspects of validity and 
reliability is one thing; proving that the test is valid is much more in- 
volved. 

Mere scores, as such, have little meaning, A candidate may achieve 
a score of 80, 60 or 90. Does this mean he is 80%, 60% or 90% proficient? 
Actually, each score represents merely a record of achievement on a sam- 
pling of acceptable occupational competence. Any judgement regarding com- 
petent r is an inference from this score which reflects only the number of 
c ° rs. Its validity is not self-evident, but is something that 

lished on the basis of adequate evidence. This evidence is 
0111 1) what an occupatL Uy capable individual needs to know 

anu what he must be able to do in the occupation, according to certain 

standards of performance and craftsmanship. Critical examination and 
judgement then form the basis for measuring knowledge and performance. 
Statistical procedures for measu^'ng content validity are not now in ex- 
istence. Content validity rests nea-cly always on some form of expert 
judgement . 



Emperical Approach 

In contrast to the logical approach, the emperical method provides 
kinds of information. The following simple example provides 
identification of whether each item on an occupational competency exami- 
nation does or does not measure individually what is expected of the test 
as a whole. By listing the test items in numerical order and calculating 
the percent of the total number of candidates who answer each one correct- 
ly, those items which discriminated and those which failed to do so may 
easily be established (Table X ). 
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Table X. 



ITEM ANALYSIS OF RESULTS OF A CARPENTRY EXAMINATION 
BY DIFFERENT CRAFTSMEN 



Item 


Percentage of Selected Groups Passing the Item 


Carpenter 


Other Construction 
Workers 


i 


90% 


85% 


2 


60% 


65% 


3 


85% 


26% 


4 


25% 


48% 


5 


76% 


30% 


200 


25% 


28% 



In this case, the written test for carpentry, containing 200 items, 
was administered to a group of journeymen carpenters and journeymen in 
other construction occupations. From these scores, it becomes evident 
that Items 3 and 5 are unique to the carpentry trade. Items 1, 2, 4 and 
200 do not discriminate between carpenters and other construction workers. 
From an empirical approach, using an item-by~item analysis, the value of 
each test item to measure specific occupational competencies may be estab- 
lished . 



III. ANALYZING PERFORMANCE TEST SCORES 
A . Steps in Making the Analysis 

1. The first step in analyzing a performance test score is t^ >nt 

test scores into a distribution. This is done to derive mo. ... uning 
from them. Actually, a test should be evaluated by asking three 
questions : 

a) Did the test discriminate? Did all the testees achieve about the 
same score or was there a variety of scores on the test? Do the 
test results show clearly as to who was the best performer, the 
next best, and so forth? 

b) How difficult was the test? The frequency distribution will re- 
veal that, if nearly all of the group achieves high scores, the 
test was too easy. When most examinees obtain very low scores, 
the test might well be too difficult. 

When the examinees' scores cover a wide range, with some individu- 
als getting high scores, some low scores, but the majority scoring 
in the middle range, the tesb^r^f^ects a proper difficulty level. 
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c) Did the test accomplish its mission? Whether a test has ful- 
filled its purpose can be quickly determined from a score distr 
bution chart (Table XI ). 



Table XI FREQUENCY DISTRIBUTION OF SCORES 



Raw Score 


Tall les 


Frequency 


8 


i 


1 


7 


ii 


2 


6 


in 


3 


5 


1111 


4 


4 


11 


2 


3 


11 


2 


2 


1 


1 




Total Number: 15 





The scores are listed from the highest to the lowest in a vertical 
column. The number of examinees who achieved each score are tallied 
in the middle column. One examinee has achieved a score of 8, four 
have scores of 5, and one a score of 2. Obviously, this test dis- 
criminates among those who took the test. 



Difficulty Level of the Test 

For a visual evaluation of the frequency distribution it is helpful 
to show the distribution graphically as a curve. This may be done by 
plotting the raw scores from the test along the horizontal axis, and the 
number of individuals taking the test alor>^ ' — a ] axis. 

(Figure 13) 



Figure 13. CURVE OF TEST OF PROPER DIFFICULTY LEVEL 
FOR MAXIMUM DISCRIMINATION 




SCORES 
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5 



20 



The bunching of scores near the middle of the scale simply shows that 
this is where most of the examinees scored. There are neither a lot of in- 
dividuals showing high competency nor a disproportionate number of less com- 
petent individuals. Generally, this kind of curve describes Lest that is 
at the proper difficulty level for the individuals being tested. 



Referring to the graph in Figure 14, it is apparent 
did not do very well in the test since most achieved low 
test was well constructed, it may mean that the test was 
the examinees were not qualified. However, as the graph 
the test was difficult for those who took it. 



that the examinees 
scores. If the 
too difficult or 
is interpreted. 



Figure 14 CURVE OF A DIFFICULT TEST 




The opposite condition seems apparent from Figure 15. Very few low 
scores were achieved as the majority of the examinees received nearly per- 
fect scores- This test was easy for the individuals tested. In order to 
discriminate better, the test items and ter ( should be made more difficult. 

Figure 15 CURVE OF AN EASY TEST 
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Th e fundamental principle of testing is that there are 
differences in individual proficiencies. The tests are to be 
constructed to obtain scores that indicate these differences. 

While these procedures provide a relatively simple and 
quick method of test result evaluation, there are other (statis- 
tically more involved) methods which are later reviewed. 



IV. DETERMINING TEST RELIABILITY 



A . Factors that Determine Reliability 

Although validity is one of the major considerations in 
assessing the quality of a test, its reliability is equally impor- 
tant. A test which is valid must, also, be reliable. Reliabi- 
lity is the consistency with which the test yields the same re- 
sults in measuring what it does measure. However, a test may be 
reliable and not valid. To exaggerate, for instance, if height 
is used to measure intelligence, it will be reliable because 
height measured with a yardstick is very constant but will not 
be valid for the purpose of measuring intelligence. Reliability 
pertains to a class of test characteristics. These are stability, 
equivalence and internal consistency. 



1 . Test and Retest Method 

In the test and retest method, the test is administered 
to a group of individuals representative of the group to which 
the test will eventually be administered. After a suitable 
lapse of time, say four to six weeks, the same test is admin- 
istered to the same group and the correlation coefficient be- 
tween the first and second administration of the test computed 
This coefficient then serves as a reliability estimate. It is 
called "coefficient of stability". 



While there are some advantages in the test and retest 
method, there are also some limiting factors. The choice of 
appropriate time interval is a critical factor. Additional 
limiting factors, such as: availability of a suitable test 
population, time of the examinees, accessibility to testing 
facilities, etc., make this method of limited practical value. 



143 






124 



2 . 



Administration of Equivalent Forms 



The equivalent form of an occupational competency examination 
avoids these disadvantages. Two forms must be constructed, so that 
all test items and the tests are as similar as possible. The group 
takes the one form of test and, as soon as possible, the other form. 
The agreement between the group results on the two tests represents 
the coefficient of equivalence and stability. This method has 
certain advantages over the Test/Retest Method. A larger sampling 
of the occupation is possible and there is no need for concern 
about memory and intervening practice. However, the same disad- 
vantages of availability of test population, limitation of examinees, 
and accessibility of facilities, make it less suited for occupational 
competency testing. 

3 . The Split Halves Method - Subdivision of a Test Into Equal Part s 

a) When neither of the two procedures previously discussed is 
feasible, it may be practical to divide the test into halves. 
Usually, the odd numbers are contained in one test for which one 
set of scores is obtained for the group for one score. The even 
items are included in the second half for another score. Thus, 
two scores, which are reasonably equivalent, are obtained from 

a single test. 

After th ‘rest is administered, two scores are obtained from the 
same test. The correlation coefficient, determined from these 
scores, reflects the reliability of the test. This estimate of 
reliability is often referred to as the split-half reliability 
coefficient or coefficient of equivalence. Since this value 
holds true for or J y one-half of the whole test, the reliability 
of the whole test must still be determined. 

b) Calculating Reliability 

The Spearman-Brown formula provides a moans r ui r- Mr fng. 4_ he 
whole test from its halve? ..w ated reliability of the 

half test is substituted in the following equation: 



Reliability of whole test = ^(reliability of half test) _ 

1 4- (reliability of half test' 



Two other split halves methods do not require computing the 
correlation coefficient. In the first method, only the standard 
deviation of each half test and of the total test are needed. 

to = — — ^ - r = correlation coefficient 

St 2 So = standard deviation of odd half 

Se = standard deviation of even half 
St standard deviation of total test 
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The second formula yields the same results and requires only the 
standard deviation of the difference (Sd) between the half test 
scores and the standard deviation of the whole test. 



If two parallel forms of the examination, written and performance, 
are constructed, the calculation of reliability is both simple 
and appropriate. The two forms of the examination are adminis- 
tered to the same group. Correlations are then calculated be- 
tween the corresponding part- scores in the two forms, as well as 
the total scores correlations, the reliability coefficients of 
the test itself, and of the part-scores. This procedure is simi- 
lar to the Gutman formula. 

) Other Methods of Estimating Internal Consistency 

Kuder £.nd Richardson do not require the splitting of the test into 
halves and rescoring and calculating a correlation coefficient. 

n = number of items in the test 
St = standard deviation of the total 
test scores 



r = 



n 



n- 1 



-St - (npq) 

St 2 



p _ arithmetic mean of test scores ^ Mt 

n n 

q = 1 - p 

This formula underestimates tbp intf »is ncy o ^ a test if 

there is v.iri 'it-T on in di * Li^u^ty items. 



Another Kuder-Richardson formula, called Kuder-- -hardson 20, re- 
quires the counting of the number of persons pa ■lag each item, 
dividing this by the number of persons taking t_h< est, subtracting 
che resulting proportion (p) from (1) to obtain , , getting (pq) 

from a table and adding the (pq’s) for all item t ... get (gpq) • 

Since (p) often valuable as part of an item ana" /sis of a test 
in computing the difficulty of each item, the labor required in 
determining (p) for each item serves more than cne purpose. 

= number of iteais on test 

standard dert^rion of the total 
test score 

= proportion oc persons passing 
each item 

= 1 - p 



r = 



n- 1 



St z -(gpq) 

__ t 2 VWJ -^ 



n 

St 
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d - Determining Reliability Through Analysis of Variance 



For the performance examination, Hoyt's methodology of determin- 
itig reliability through analysis of variance is suggested. The 
three sources of variance would be: (a) the difference between 

the mean scores of individuals in the group, (b) differences be- 
tween the mean of the two forms, and (c) an error component. 

A- B 

reliability = — g— A = mean square (difference between 

individuals) 

B = mean square (error) 

Another form of the same approach is the next formula: 




1. The difficulties or percentages of correct items - P 

2. Tne correlations between items and the total test 

score - ret 

3. The standard deviation of the test c r t 



~_FP.q 

2 



This formula should be applied if an item analysis study has 
been made that provides the item test correlation coefficient 
and the difficulty value for each test item. 



V. DIFFICULTY LEVEL OF TEST ITEMS 



re = 



cr t 



2 - Spq 



2 V 




+ pq 



+ 



2ov 



A . The Difficulty Index 



The difficulty level of a test item provides some indication of the 
extent to which it does its job of measuring competency. The power to 
differentiate between levels of trade performance and knovvledge is char- 
acteristic of a good test. For occupational competency test developers, 
fchis represents a difficult challenge because of the range of occupations 
In some, the skills constitute the major portion of competence; in others 
the more formal information and technical know-how determine proficiency. 
In still other trades or technical occupations, there is somewhat of a 
balance between manipulative skills and occupational information. A 
measure of difficulty level for test items may be obtained from the data 
for the combined high and ^arj|groups of scores achieved on a test. 
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The difficulty index is computed 
dates answering the item correctly by 
low groups (high group usually within 
within the lower 27%). Example: if 

group with 12 passing the item and 8 
difficulty index is: 



by dividing the number of candi- 
the total number in the high and 
the upper 27 L and the low group 
there are 16 candidates in the high 
in the low group passing, the item 



Formula for Difficulty Index; 



Difficulty 

Index 



Number of items answered 
correctly in high group 

Total number within the 
high group 



Difficulty 

Index 



12 + 8 

16 + 16 



3SL 

32 



.625 



Number of items answered 
correctly in low group 

Total number within the 
low group 



The smallest possible value that 
largest 1. The larger the difficulty 



can be obtained is zero and the 
index, the easier the item. 



Difficulty Distribution 

bearinSTon^ts^L 015 T ° ccu P ati ° na 1 competency test has an important 
ring on its value. As a general rule, the average difficulty of th<= 

I 5 ,; 6 " “ * test correspond to the average compft ency^? 

craftsmen in the occupation, i.e., the items should be such that on the 
verage about half the candidates will answer the test items correctly 
m-wever, xn occupational competency testing, it is desirable to test 

£r S pread 8 ove^ the f ka ° wled S e > th * difficulty level of which must 

oe spread over the range within which it is desired to differentiate 
It should tend to cluster around 50% for the group. 

formula f imPler Version> adequate for most purposes is the following 

Difficulty Index = x _ Total Right 

N Total Number 

high-medium- low d re ^“f re dividing the group into quartiles of 

xgh medium low, where the median consists of 50% of the candidates. 



VI. DISCRIMINATION OF TEST ITEMS 
Discrimination Index 

tke difficulty level of an item determines, in part the 
1 °i th ? teSt ltem to discriminate between the proficiency of 

of 1 50 ’b th f d l0W group pass ' the item "'would have a"dif I icu 1 ty"i n dex 
of .50, but would not discriminej^ [P etween those who did wall end tholj 
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who did poorly on the test as a whole. On the other hand, if 16 of the 
high group passed and none of the low group, its difficulty would, also, 
be .50. This might lead to the conclusion that the test had maximum 
discriminatory power. 



A measure of discriminatory power may, also, be obtained on the 
basis of high and low groups of the wrong answers. The discrimination 
index may be computed by counting the wrong answers in the low group 
(VI'i) , and subtracting the wrong answers in the high group (W^) , and di- 
viding by the number of candidates (N) in eigher the high or low group: 



Discrimination Index: G 



w x - w h 



N 



The discrimination for a group of 8 low and 8 high group scorers 
among a total group of 16 equals zero. If the low group was 16 and the 
high group zero, then the discrimination factor would be 1. When there 
are more wrong answers in the high group than in the low group, the 
discrimination factor becomes negative. 

The range of values of the discrimination index depends upon the 
difficulty of the item. Items with difficulty levels of zero (0) or 
one (1) always have discrimination indices of zero. They cannot dis- 
criminate between craftsmen, of low or high proficiency. The maximum 
discriminatory power drops as the item difficulty departs frcm ,50. 
Since test makers strive for tests that have high discriminatory power, 
most test items should have difficulty levels between .40 and .60. 



VII. LENGTH OF TESTS AND RELIABILITY 



A. Reliability Coefficient and Length of Test 



The reliability coefficient may be affected by the length of the 
test, the occupational proficiency range of the individuals to be 
tested, and the conditions under which the tests are administered. 
Other factors being equal, the reliability of a test is a function of 
its length; longer tests tend to be more reliable than shorter ones. 
The more samples taken of a certain area of skill, knowledge and/or 
behavior, the more reliable will be the appraisal of that area. 



However, there is a limit beyond which a test of reliability is 
no longer practical. In using the formula: 



rr,n 



nr 

1 + (n~l)r 



for a test of 50 items with a reliability coefficient of .80, the 
length of the test could be doubled to 100 items and raise the whole 
test reliability coefficient to ,888, tripled to 150 items for a coeffi- 
cient of .923 or quadrupled to 200 for a coefficient of .941. Note, in 
each instance, that the reliability coefficient increment from 50 to 
100 items (.088) kept decreasing 100 $0>jj.5O (-035) to 200 items (.018). 
Eventually, this would level off. 

148 



129 



B. 



Establishing Lengths of Tests 



Should it become desirable to learn hew much longer a test of 
known reliability should be made to achieve a particular reliability, 
the following Spearman- Brown formula may be used: 



n 



r nn (l"* r n) 



11 



< 1 ‘ r „n> 



By properly substituting the values for r^-, and r nn in this equa- 
tion, the multiple by which the number of test items n should be in- 
creased can be obtained. 



r nn . 90{desired reliability), ~ *70 (test reliability) 

n - -^gLQ- , (1-70 ) , . (j. 90) (.30) = 3 . 36 

.70 (1-90) (.70) (.10) 

Example: A test of 50 test items has a reliability coefficient of 

.70. The question is, how many items should be included to 
establish a reliability coefficient of .90? Substituting 
values in the formula, the increase would be 3.86 x 50 = 

193 test items. 



VIII. RANGE OF ABILITY AND TEST SCORES 

The reliability coefficient depends in part on the range of ability 
for the group for which the reliability coefficient is computed. A relia- 
bility coefficient for a test, based on a highly selected group of compe- 
titors, will be lower than the reliability based on an ur selected , more 
variable group. Although the reliability coefficient may vary considerably 
with the range of ability, the standard error of the score may not be ex- 
pected to do so. If it is assumed that the standard error of the scores is 
constant throughout the whole range of scores, it can be shown that the fol- 
lowing relationship holds: 

R = 1 51- ( 1 ~ r ) 

6 2 

r= reliability coefficient for a group of test scores with standard 
deviation 

R= reliability coefficient of the same test for a group of scores with 
a standard deviation 

If a reliability coefficient of .80 has been obtained from a group of 
candidates for which the standard deviation if 10, then to find the esti- 
mated reliability for a group of subjects for which the standard deviation 
is 20 , 

149^ = i_ 100 (1-.80*) = 95 
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IX. RANGE OF ABILITY REGARDED AS STANDARD 



No data is available for occupational competence tests which Indicates 
a minimum coefficient of reliability. Among test specialists, the general 
opinion prevails that a value approximating .90 represents an acceptable 
standard . 



X. ITEM ANALYSES OF TEST F ESULTS 

The total value of a test depends upon the sum total of the individual 
test items. For the test constructor, it is important to know: 

A. What contribution each test item makes to the total test score, 

B. How well the test scores discriminate in measuring proficiency 
among more or less competent individuals, 

C. Do the options in the mult iple- choice test items attract fairly 
well distributed responses, 

D. Are there some choices which have been ignored by nearly all 
examinees, so that they might as well be excluded. 

In the formulas that have been developed for establishing difficulty 
and discrimination data, test results for high, medium and low groups have 
been used. The high group constitutes 27% of the tested group, the low 
group 27%, and the remaining 46% represents the medium or average compe- 
tence . 



A typical item analysis Is designed to provide the test constructor 
with information about the contribution of each test item to the total 
test score. Table XII shows an Item Analysis of Test Results. The ratio- 
nale is that persons who score higher on the test, as a whole, should 
score higher on each test item than persons who do poorly on the test as 
a whole. Items 1 and 2 in Table XII reflect these expected differences, 
but items 3, 4 and 5 and 200 do not. The latter items should be reviewed 
by the t esc constructor to determine the cause of these differences. 



Table XII. ITEM ANALYSIS OF TEST RESULTS BY CANDIDATES 
WITHIN SAME OCCUPATION 



Item 

Number 


Percent Passed 
Upper 27% 


Percent Passed 
Lower 27%, 


Discrimination 

Index 


Difficulty 

Index 


i 


.80 


.40 


.40 


.40 


2 


.90 


.20 


.70 


.45 


3 


.36 


.50 


- .40 


.57 


4 


.66 


.60 


.06 


.47 


5 


.98 


.98 

\ \ 


.00 

- 1 I 


.02 


200 


.40 


00 1 1 
o 


- .40 


.40 
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The objective of item analysis is to improve the measuring instrument. 

It is possible to identify ambiguities, misleading words, totally irrelevent 
items' and other errors or weaknesses. 



XI. ANALYSIS OF ALTERNATE CHOICES 

If the alternatives to the correct choice are obvious or are ignored by 
the examinees for other reasons, the test item loses its value to the overall 
test. The following examples demonstrate the steps in analyzing a multiple- 
choice test item: 

Example 1: On a 100- point test, candidates made scores of 95-92-91, the 

other 22 candidates achieved scores ranging from 49 to 67. The 
measure of central position apt to be best for this group of 25 
students is: 

A - The Mode 
B - The Median 
C - The Mean 

D - An Average of Mean and Mode 
The responses to the choices are listed in Figure 16. 

Figure 16. ANALYSIS OF EXAMPLE 1 



Response 


Choice 


Difficulty 


D i s c r imina t i on 


A 


B 


c 


D 


Upper 27% 


i 


17 


- 


- 






Lower 27% 


5 


9 


3 


1 


0.72 


.44 




6 


26 


3 


1 







This item is relatively easy. Twenty- six students, out of 36 in the com- 
bined high and low group, marked the correct choice. 

The difficulty index is calculated as 26 _ 

36 " 

The discrimination index is calculated as 8 

18 “ - 44 

Three distractors attracted, at least, one member of the lower group. 

All are somewhat plausible to those who do not know the correct answer. This 
item might well be placed at the beginning of the test. 
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Example 2: 



In an oscillator, which uses a pie 20-electric effect for fre- 
quency control, energy is fed back to the grid circuit by 

A. a tapped coil 

B. inter- e lectrode capa.city 

C. a capacitor between plate and grid 

D. inductive coupling 

E. none of these 



Figure 17. ANALYSIS OF EXAMPLE 2 



Response 


Choice 


Difficulty 


Discrimination 


A 


B 


C 


D 


E 


Upper 27 % 


12 


3 


- 


2 


i 






Low^r 27 % 


4 


2 


4 


6 


2 


.44 


.44 


_ 


16 


5 


4 


8 


3 








This item appears to be more difficult than item 1 because fewer than one 
half of the examinees, in the combined high and low group, marked the key 
answer (A). The difficulty factor is .44, and the discrimination index is 
.44. The maximum value of discrimination is 18-2 or .89. 

18 

While this is low in discrimination, it distinguishes between high and low 
groups. item 2 is closer to .50 in difficulty, and the distractors seem to 
be effective, particularly, for the low group. 

Example 3: The relationship between the pulse repetition and the period is: 

A. the higher the pulse rate the longer the period 

B. the higher the pulse rate the shorter the period 

C. the lower the pulse rate the shorter the period 

D. none is correct 



Figure 18 ANALYSIS OF EXAMPLE 3 



Response 


Choice 




Discrimination 


A 


’ B 


c 


D 


Difficulty 


Upper 27 % 




3 


- 


15 


.33 


- .33 


Lower 27 % 


- 


9 


- 


9 




- 


12 


- 


24 
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Obviously, there is a problem with this item. The item, in its present 
form seems quite difficult. One- third of the group marked the item cor- 
rectly. The proportion of correct answers is larger in the lower group 
Therefore, it would seem easier, or it was confusing to the higher group. 
Options A and C were not selected by anyone , 

The item was revised as follows: 

Which statement describing the relationship between the pulse repetition 
and the period is correct? 

A. the higher the pulse rate the longer the period 

B. the higher the pulse rate the shorter the period 

C. the lower the pulse rate the shorter the period 

D. the relationship is not apparent 



Figure 19 ANALYSIS OF REVISED EXAMPLE 3 



Response 


Choice 


Difficulty 


Discrimination 


A 


B 


c 


D 


Upper 27% 


- 


18 


- 


- 






Lower 27 % 


2 


12 


2 


2 


.83 


.33 




2 


30 


2 


2 







The item has become relatively easy and discriminates reasonably well 
The distractors are functioning. 

These examples show one approach which can be used for test improvement. 
The statistical tools are of substantial help in test construction. They can 
point to weaknesses and point up the direction in which to proceed. Occupa- 
tional competency testing involves many variables which are difficult to fit 
into a definite statistical pattern. Analysis and careful definition of the 
objectives, which lead to more objective criteria, are still an art. Eventu- 
ally, they may become a science. 



XII. TEST STANDARDS AND NORMS 



A. Establishing Norms 

A test may meet many or all the statistical requirements. The tabu- 
iated results show an acceptable spread among the individual scores, and 
the difficulty and discrimination coefficients fall into acceptable 
ranges. By themselves, these scores have little meaning. For, at what 
point, in the tabulated spread of scores, is the dividing line which sep- 
arates the occupationally competent individual from the one whose skill 
and knowledge is not at a level acceptable for teaching others. In short, 
at what point does a candidate pass or faj. 1 ? 
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A type of measure that assists in the interpittat^nn of test scores 
is the norm. According to Adkins, 

"Norms are a measure of a specific function based on a specific 
sample that provides standards against which particular test 
scores can be interpreted." 

The establishment of norms facilitates the setting of "pass" or 
"fail" levels. Such levels can be developed in measures of central ten- 
dencies, expressed in stand; deviations, and in percentiles or stanines. 

Assuming that the tests are truly representative of the competence 
of an occupation, a cutoff point at the 50th percentile would assure, at 
least, average proficiency. For more exacting standards, a cutoff point 
could be established at the 70th or 75th percentile. In such a situation, 
the candidate would be among the top quarter of the workers in a particu- 
lar occupation with respect to occupational competence. 

Such an arrangement would allow for variations among states and trade 
and industrial/ technical teacher training institutions. For some occupa- 
tions, certain subdivisions could be established with percentile ratings 
to provide greater flexibility in evaluating an individual’s range of com- 
petence. To use occupational competency tests under varied conditions and 
requirements, norms must be established against which test results can be 
eva luat ed . 

B . Types of Norms 

Preference in achievement testing has been given to percentile norms. 
A number of other methods can be employed for a variety of purposes. It 
must be kept in mind, at all times, that rax/ scores achieved on a test are 
of little significance unless they indicate a degree of proficiency which 
can be related to some level of accepted proficiency in the occupation. 

1 . Computing Percentile Norms 

Percentile norms are fairly easy to interpret. They express 
rank in terms of common units and can be computed by the 
following formula: 

P = L + J£N-X_ .1 
P 

L = bottom limit of the interval continuing the percentile 
p = is the percentage of cases below the percentile 

F = is the number of cases in all intervals below the interval 
containing the percentile 

fp= is the frequency within the interval containing the 
percentile 

N ~ number of frequencies 
1 — is the size of the class interval 

Computing each of the 100 scores by formula is rather tedius. This 
computation can be obtained by computor or, more readily, by arithmetical 
or graphic interpolation from a smaller number of locations. 

O 
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A more satisfactory method is to determine the percentile value of the top 
of each class interval by finding what percentage of the cases are covered by 
the cumulative frequency to the top of each interval and then to interpolate 
graphically for intermediate values. Table XIII illustrates the data needed 
tor Lhe second method. 

The frequencies (f) in the second column are < "iulative frequencies 
(column three) by successive addition from the lowest to the highest scores. 
The cumulative frequencies are converted to cumulative percentages by multi- 
plying each entry by the reciprocal of N. These cumulative percentages (cumu- 
lative percent) are the percentiles correspondin' to the tops of each score 
interval . 



Table XIII. COMPUTATION OF PERCENTILE SCORES 



Scores 


(f) Frequencies 


Cumulative Frequencies 


Cumulative Percent 


69-71 


4 


200 


100 


66-68 


6 


196 


98 


63-65 


8 


190 


95 


60-62 


12 


182 


91 


57-59 


16 


170 


85 


54-56 


18 


154 


77 


51-53 


22 


136 


68 


48-50 


30 


114 


57 


45-47 


24 


84 


42 


42-44 


18 


60 


30 


39-41 


14 


42 


21 


36-38 


12 


28 


14 


33-35 


8 


16 


8 


30-32 


4 


8 


4 


27-29 


4 


4 


2 




200=N 





For greater convenience, accumulated percentiles may be plotted graphi- 

call y on a percentile curve. Then, percentiles can be read directly without 
computation. 



f = frequencies 

Accumulated frequencies 4+4+8+12= 28 

Percentiles _ accumulated frequenc ies 28 

For scores 36-38 number of candidates 200 
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Figure ?Q PERCENTILE GRAPH 
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To find the score for any desired percentile P, start from the P value 
on the horizontal axis, determine where a vertical line for this point cuts 
the vertical axis. 



Percentile ranks provide very unequal units along the scale of perfor- 
mance on a test. The difference level in performance between scores at the 
50th and 55th percentile is less than between scores at the 90th and 95t 
level in a bell- shaped diotribution of raw scores, even though the differences 
in percentile rank are equal. Despite these disadvantages, percentile ranks 
are so easy to understand that they are used as norms for many standard tests. 
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2 . Standard Scores 



a • Z Score Defined 



A standard score or 2 score is defined as the dc fat 
the arithmetic means in standard deviatio units. T e i, 
such scores is: 



of a score from 
ula for calculating 



2 = 



x - m 



o Z = standard score 

x = any raw score 
m = arithmetic mean of raw scor 
s = standard deviation 

If the mean of a distribution is 100 and its standarc deviation 20 a 
candidate with a raw score of 120 receives a 2 score of 1.00. 



2 = 



120 - 100 
20 



= 1.00 



The advantage of the standard score, over the percentre rank for ex- 
pressing the relative standing of an individual in his gro. , i s that stan- 
dard score units ate equal over the entire scale so that small differences 

at one point on the scale mean exactly the same thing as an equal difference 
at some other point. 

, The , chief valu ® of standard scores is that they express scores in conven- 
ient and common units so that achievement on several tests, written and per- 
formance, can be understood and compared without reference to the original 
units. 

Z = 



jO - 100. _ 10 _ o 50 

20 "20 



*Sr d n Td a C ?K e - have ^ he useful characteristics that their mean is always 
arithmetic standard deviation equal to 1. All raw scores below the 

Several “ !!" (llk< ff he example with a raw score of 90) become negative. 
Several variations, like the derived score, have been introduced to eliminate 
this inconvenience. 



D 




+ 50 



A derived score 
The formula changes 



(D) is simply defined as 10 times the Z score plus 50. 
the mean (m) to 50 and the standard deviation (s) to 10. 



b. 



Differences Between Standard Scores and Percentile; 



n .__^ ^stinct difference exists between the standard score method and the 

Jhl S n ii le h ° d * The standard scores are distributed more norma' iy along 
the bell curve; the percentile scores tend to concentrate at the - t ean. Per- 
centile scores cannot be averaged, thd ’’various standard scores can It is 
of course, an advantage to be able to average levels of performance when it 
1S desired to average the results of several tests. 
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3. Normalized Standard Norms 



The major features of percentile scores and standard scores can be 
combined into normalized standard scores by taking the following 
steps: 

a) Convert raw scores into percentile scores 

b) Convert percentile values into ^ values. — values are found in 
special tables ox normal curve relationships which give values 
of (x) or distances from the mean which include (o) (specified 
proportions of the area or percentages of the cases) . 

Example: Determine the normalized standard score corresponding 

with a raw score of 47 . 

Percentile equivalent 80 is to be expressed in the standard scGre. 

In a normal distribution, 50% of all cases fall below the mean score 
.-50 is subtracted from 80--- -50 = 30. Thirty percent of percent- 
age cases fall between mean and percentile score of 80 in a normal 
distribution. The value of x 7 o for an area of 30 is, approximate- 
ly, .84. This value indicates the point on the base line of a normal 
curve which separates the lower 80 percent from the upper 20 percent. 

Thus, .84 is the normalized standard score corresponding to a raw 
score of 47. 

For raw scores corresponding to percentiles below 50, the standard 
score will be negative. A negative sign merely indicates that the 
distance referred to is below the mean. 

In the determination of standards for a test, the conversion of one 
score to a corresponding score on another test must be kept in mind. 
Such conversions can be achieved by the use of special charts. 

4 . T- Scores 



The T- scores serve the same purpose as the Z score. T- scores are 
based on the same principle, are always positive, and express larger 
units. Thus, T-scores eliminate the need for dealing with negative 
numbers and decimal fractions. A T- score is computed with the 
following formula: 

T = 10 Cx ~ m) 4-50 x = any raw score 

s m = arithmetic mean of raw score 

s = standard deviation of raw score 

A T- score of 60 means a score of one standard deviation above the 
mean, a score of 70 two standard deviations above the mean, and so 
on. T-scores of 40 and 30, similarly, indicate scores at one and 
two standard deviations below thpcjnean, respectively. 
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Chapter VIII 



NATIONAL OCCUPATIONAL COMPETENCY TESTING PROGRAM: 

A NATIONAL URGENCY 

I. THE NEED FOR A NATIONAL OCCUPATIONAL COMPETENCY TESTING PROGRAM 

The key to the success of vocational education programs, at every level, 
is the occupationally competent teacher. The need for aualified tradesmen 
and technicians for teaching positions in vocational education programs is 
critical. 



Determining occupational competency requires special personnel, is time 
consuming and expensive. Lack of such personnel and limited financial re- 
sources place a heavy burden upon those states which recognize the need for 
a sound testing program. In the past, personal interviews, review of em- 
ployer's recommendations, oral examinations by experienced administrators and 
teacher educators have been used with some degree of success to select 
tradesmen and technicians for teaching positions. With the number of teachers 
required for the expanding vocational programs, objective information is 
needed which represents evidence as to an individual's competence. 

Well designed and validated tests provide accurate and reliable infor- 
mation from which to evaluate an individual's occupational proficiency. The 
results of such tests can then be the basis on which to admit a candidate to 
an industrial teacher education program, provide the evidence required for 
certification, and enable institutions of higher learning to grant advanced 
standing for occupational proficiency in degree programs. 

In this project, it was established that there are twenty centers, or 
state bureaus, which devote part of their effort to developing and adminis- 
tering competency tests. The survey further revealed a great deal of dupli- 
cation of effort. 



With the exception of a few states, limited evidence is available that 
the occupational competency t.ejts have been subject to validation and reliabi- 
lity studies, or that standards or norms have been established; in short, 
that they are accurate and reliable instruments. It is, therefore, not sur- 
prising that state officials, as well as industrial teacher educators, have 
expressed serious concern for better and more accurate instruments. 

-*- r ' summary, it is obvious that objective, reliable and valid tests are 
more than ever necessary to measure occupational competence for institutional 
teaching, teaching within industry, within the military, and so forth. It is 
equally apparent that continued limited scattered efforts by the states can 
not produce the kinds of instruments needed. 



The 
national 
in secur 



solutions to occupational competency test are to be found through a 
coordinated effort uqflex < a broadly administered system, with built- 
ity and flexible controls that meet the needs of each state. 



159 



II. ADVANTAGE OF A NATIONAL TESTING PROGRAM 



While a national effort must overcome a number of problems, the advan- 
tages well outweigh its disadvantages. Complementing the seven major and 
other additional functions defined in Volume I of this study are the follow- 
ing values for a National Testing Program: 

A. Tests can be prepared more economically, thus conserving limited 
financial resources. 

B. A pool of latest and most objective methods of test development 
can be applied to the preparation, administration, and evaluation 
of occupational competency tests. 

C. Personnel now devoting part-time on a " hit-and-miss” basis can be 
relieved of test development responsibility to devote full-time to 
other program purposes. 

D. Central data processing facilities can be more effectively utilized 
to evaluate tests and test results, and the techniques of statisti- 
cal analysis may be more efficiently applied to create instruments 
that are accurate and reliable. 

E. Tests can be more comprehensive in scope, yet provide greater flex- 
ibility to permit evaluation on various levels of competence within 
an occupation. 

F. Tests, administered cn a national basis, provide the information 
for the establishment of norms and standards essential for teaching 
in vocational programs. 

G. Tests would be available to the states to use and interpret ac- 
cording to their own needs and certification requirements, and 
with built-in safeguards for state controls. 



III. THE CONSORTIUM OF STATES AS A FACTOR IN RAISING STANDARDS 
AND RECOGNITION OF VOCATIONAL TEACHERS 

A Consortium of States could establish an effective and efficient base 
for a National Occupational Competency Testing Center and Program. It could 
provide an administrative vehicle, develop administrative policies, set es- 
sential controls acceptable to each state, and lay out plans for implementa- 
tion. Most important of all, bv combining resources, evaluative instruments 
of high quality and acceptability could be developed which would: 

A. Establish certain minimum required levels of competency; 

B. Provide reciprocal arrangement, when all requirements for certifi- 

cation are met, for acceptance of individuals from one state to 
another; f ■'* 

C. Develop occupational competence levels among teachers on a broader 
national basis; 
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B. Encourage (through the granting of extensive content collegiate 
credits) vocational teachers to engage in studies in higher edu- 
cation leading to supervisory and administrator certificates; 

E. Gain greater acceptance of occupational competency in institutions 
of higher learning; and 

F. Establish a more professional approach toward certification and 
licensing of vocatioiral teachers and gain greater professional 
recognition. 



For the future vocational teacher, the successful passing of an occu- 
pational proficiency examination acknowledges that the individual tradesman 
or technician has the qualifications to function in one of the most important 
prof essions-- that of a teacher. 
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A. Selected Materials Arranged According 
to Sequence in Chapters 



B. Selected References Arranged Alphabet- 
ically 
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APPENDIX A 



SELECTED MATERIALS ARRANGED ACCORDING TO SEQUENCE IN CHAPTERS. 

1. Cole, Lawrence F. and Bruce, William F. 

Educational Psychology , pp-624-630 World Book Company, 
Yonkers on Hudson, N. Y. 1950 

2. Fischer, George 

S ^ a ^ e Boo Hs of the Reverend George Fischer in England (1864) 
with their "standard specimens" with which specimens of 
pupil work could be compared; also, the provocative article 
in Th e _F orum in 1890' s by its editor. Dr. J. M. Rice, who 
had measured thousands of American school children in 
spelling and other subjects by a series of tests. 

3. Shimberg, Benjamin 

The Performance Phase of Trade Competency Examination. 
Unpublished paper. Educational Testing Service, Princeton 
N. J. 1966 ’ 

4. Cole and Bruce 

op. cit. pp 130 

5. Chapman, J. Crosby 

Trade Tests p. 24, Henry Holt & Company, New York, N. Y. 



6. United States Army 

Performance Test Construction, United States Army Signal 
School; Educational Research Division, Office of Plans and 
Operations, Ft. Monmouth, N. J. 1966 



7 . Chapman 

op. cit. p. 24 

8. Thompson, L. E. , Lawrence, D. C., Allardyce, A. 

Interview Aids and Trade Questions for Employment Offices, 
Harper & Brothers, New York 1936 



9. Chapman 

op. cit. p. 24 

10. Chapman 

op. cit. p. 88 

11. Stead, William and Chartle, Carol L. 

Occupational Counselling Te c hniques Their Development and 
Application. American Book Company, New York, N. Y. 1940 

12. United States Army, Performance Test Construction, Educational 

Research Division, Office of Plans and Operations, United 
1966 GS ArmS Slgnal Corps School, Ft. Monmouth, N. J. p. 13 

163 



144 



13. Hams, C. J. (ed.) 

The Encyclopedia of Educational Research, McMillan, 3rd ed.. 
New York, N. Y. I960 

14. Krathwohl , D. R. , Bloom, B. S., and Masia, B. B,, 

Taxonomy of Educational Objectives . Affective Domain. 

David McKay, New York, N. Y. 1960 

15. Simpson, Elizabeth Jane 

"The Classification of Educational Objectives", Illinois 
Teacher of Home Economics 1966 10(4) 110-114 

16. Moore, Maxine Ruth 

The Proposed Taxonomy of the Perceptual DomaLn and Some 
Suggested Applications , Educational Testing Service, 
Princeton, N. J. 1960 

17. Krathwohl, et. al. 

op. cit. 



18. Adkins 

op. cit. p. 171 

19. Lindeman 

Educational Measurement , Scott, Foresman and Company, 
Glenview, 111. pp..63 1967 



20. Adkins 

Illustrations derived from Adkins, op. cit. p. 33 

21. The University of the State of New York 

Improving the Class Room Test , The State Education 
Department, Bureau of Test Development, Albany, N. Y. 1964 



22. Lindeman 

op. cit. pp. 78 

23. United States Navy Procedures for Constructing and Evaluating 

Achievement Examinations. Unpublished paper. Service 
School Command, United States Naval Training Center, Creat 
Lakes, 111. I960 

Adkins, op. cit. p. 41 

24. Remmers, H. H. , Gage, N. L. and Rummel, J. Francis 

A practical Introduction to Measurement and Evaluation, 
Harper and Row, New York, N. Y. 1965 



25. Michaels, W. J. and Karnes, M. R. 

Measuring Educational Achievement , McGraw-Hill Book Company, 
Inc., New York, N. Y. p. 101 1950 



26. United States Navy 
op. cit. 



145 



27. Travers, M. W. 

How to Make Achievement Tests , The Odyssey Press, New York, 
N. Y. p. 101 1950 



28. Michaels 

op . c i t . 

29. Lindquist, E. F. (Editor) 

Educational Measurement , American Council on Education, 
Washington, D. C. , George Banta Publishing Company, Menasha, 
Wise. p. 214 1951 

30. United States Army Performance Test :Construction , Educational 

Research Division, Office of Plans and Operations, 

Ft. Monmouth, N. J. p. 17 1960 

31. United States Army, op. cit. 

Federal Aviation Administration, Designing Written Achieve- 
ment Tests . Department of Transportation, Training Bulletin 
103, FAA Academy 1965 

32. Committee on Test Standards, AERA, and Committee on Test Standards, 

NCMUE, Technical Recommendation for Achievement Tests , 
National Education Association, Washington, D. C. 1955 

Technical Recommendations for Psychological Tests and 
Diagnostic Techniques, Supplement to Psychological Bulletin 
1954, No. 2, Pt. 2 

33. Lindeman, Richard H. 

Educational Measurement . Scott, Foreman and Company, 
Glenview, 111. p. 37, 1967 

34. Reminers, H. H. ; Gage, N. L. ; Rummel, J. Francis 

A Practical Introduction to Measurement and Evaluation , 
Hopper and Row, New York, N. Y. p. 120 1965 

35. Impellitere, Joseph T. 

"Constructing Valid Occupational Competency Examinations", 
Unpublished paper in Feasibility of Providing Trade and 
Competency Examinations. Final Report. Rutgers, The State 
University, Graduate School of Education. Department of 
Vocational/Technical Education, New Brunswick, N. J. 1967 

36. United States Army, Performance Testing . Enlisted Evaluation Center, 

Ft. Benjamin Harrison, Indianapolis, Ind. p. 9-1 1969 

37. Gutman, L. 

"A Basis for Analyzing Test - Re- Test Reliability", 
Fsychometrica , 1945, 10: 255-282 

38. Rulan, P. J. 

"A Simplified Procedure for Determining the Reliability of 
a Test by Split- Halves" , Harvard Education Review, 1934, 

9: 99-103 ~ 




146 



39 . 



i 

[ 

f 



i 




Kuder, G. F. and Richardson, M. W. 

The Theory of the Estimation of Test Reliability 
Psychometr ica , 1937 2:151-160 



40. 


Remmers 


, et„ al, 
op. cit. p. 


129 




41. 


Hoyt , C 


. T. 

Test Reliability Estimated 
Psychometrica , VI, 1941 


by Analysis of Variance 
pp. 153-160 


42. 


Lindeman 

op. cit. p. 


89-91 




43. 


Remmers 


, et. al. 
op. cit. }>. 


131 




44. 


Adkins , 
Bridges 


C. Dorothy; 

, Claude F- ; 
Construction 


Primoff, S. Ernest; McAdaa , Harold L. ; 
Foree, Bertiam 

and Analysis of Achievement Tests, 






Superintendent of Documents 
Office, Washington, D. C 


, U. S Government Printing 
1947, p. 145-160 


45. 


Impellitere 

op. cit. 






46 . 


Lindemnn 

op. cit. p. 


92 




47. 


Adkins , 


et. al. 
op. cit. p. 


137 




00 


Adkins , 


et. al. 
op. cit. p. 


137 




49. 


Adkins, 


et. al. 
op. cit. p. 


140 




50. 


Walker, 


Helen M. 

Elementary Statistical Methods, Henry Holt & Company, 
New York, N. Y. 1943 p. 136-138 




Adkins 7 


et. al, 
op. cit. p. 


141 




51. 


Walker 


op. cit. p. 


338-339 






Kelley, 


Truman, 

Statistical 


Method, MacMillan Company, New York 1923 






p. 373-380 


* r * „ 4 






Guilford, Joy Paul 

Psychometric Methods, McGraw-Hill, New York, N. Y. 1936 

p. 530-534 



166 



52. L indeman 



op. cit. p. 133 



Remmers 



Adkins 



op. cit. p. 61 
op. cit. p. 142 



53. United States Department of Labor 

Occupational Licensing avid the Supply of Non-Professional 
Manpower 



Manpower Administration 



Manpower Research Monograph #1 

Superintendent of Documents, United States Government 
Printing Office, Washington, D. C. 1969 p. 1 



167 




. 1(48 



APPENDIX B 



II. SELECTED REFERENCES ARRANGED ALPHABETICALLY 

Adkins, Dorothy C., Ernest S. Pirmoff and Harold L. McAdoo, Claude F. Bridges 
and Bertram Forerr, Construction and Analysis of Achievement Tests, 
Superintendent of Documents, U. S. Government Printing Office, Washington, 
D. C. 1947 

Askren, William B. (Ed.), Symposium on Reliability of Human Performances in 
Work. Annual Convention of the American Psychological Association, 

U. S. A. F. - AMRL Technical Report, No. 67-68-45 pp . 1966 

The A. P. A., AERA, NCME Joint: Committee, Standards for Educational and 

Psychological Tests and Manuals , American Psychological Association, 
Washington, D. C. 1954 

Ana s t a s i , Anne , The Con c ept of Validity in the Interpretat io n of Test Scores . 
Education Psychological Measurement 1950-10-67-68 

Anastasi, Anne, Psychological Testing , (2nd Edition), McMillan Company, 

New York, N. Y. 1965 

Baldwin, Thomas S., The Development of Achievement Measures for Trade and 
Technical Education . Office of Education Grant- 0EG2- 7000517-0585 , 
University of Illinois, Urbana, 111. 1968 

Bloom, B. S. (Ed.), Taxonomy of Educational Objectives: Handbook I, 

Cognitive Domain . Lougmans, Green and Company, N. Y. 1956 

Bloomers, P and Lindquist, E. F., Elementary Statistical Methods in Psycho- 
logy and Education . Houghton Hifflin, Boston, Mass. 1960 

Buros, Oscar Krison, The Sixth Mental Measurement Yearbook . The Gryphon Press, 
Highland Park, H. J. 1961 

Chapman, J. Crosby, Trade Tests , Henry Holt & Company, New York, N. Y, 1921 

Cronbach, L. J. , Essentials of Psychological Testing , Harper, N. Y. 1960 

Digitek Corporation, Fair less Hills, Penn. 

Fattu , N. A. and Medley, D. M. , Rationale and Construction of the K- System 

Performance Test . An unpublished report of the Air Research and Develop- 
ment Command, Lowry Air Force Base, Denver. Colo. 1961 

Gohem, Howard W. , and Karuck, Samuel, Test Construction , U. S. Civil Service 
Commission, U. S. Government Printing Office, Washington D. C. 1951 

Fleishman, Edwin A. , Performance Assessment on an Emperically Derived Task 

Taxonomy. Human Factors Institute for Research, Washington, D. C . 1967 



168 

149 



Furst, Edward J., Constructing Evaluation Instruments , Longmens, Green & 
Company, New York, N. Y. 1958 



Fear, Richard A. and Byron, Jordon, Employee Evaluation Forms for In terviews, 
Psychological Corporation, New York, N. Y. ” 

Green, Helen James, A Manual for Selected Occupational Tests for the Use 

in Public Employment Offices , Bulletins of the Employment Stabilization 
Research Institute, University of Minnesota, Vol. II, No. 3, July 1933 
Minneapolis, Minn. 1953 

I. B. M. Corporation, Binghamton, N. Y. 

Krathwohl, David R. , Bloom, Benjamin S. and Masia, Bertram B. , Taxonomy of 
Educational Objectives Classification of Goals, Handbook II Affective 
D omain , David McKay Company, Inc., N. Y. 1964 

Kynard, Alfred Tennyson, Criteria for Determining the Value of Work Experience 
for Teachers of Trade and Industrial Education , University of California, 
Los Angeles, Calif. 1960 

Lindeman, Richard II., Educational Measurement , Scott, Foresman and Company, 

New York, N. Y. 1967 

Lo.fgren, Paul V. W. and Allen, David, An Abridgement of Study of the Trade- 

Technical Teacher Proficiency Testing Program in California , Unpublished 
paper. Division of Vocational Education, University of California, Los 
Angeles, Calif. 1969 

Measurement Research Center, Iowa City, la. 

Moore, Maxine Ruth, A Proposed Taxonomy of the Perceptual Domain and Som e 

Suggested Applications , Educational Testing Service, Princeton, N. J. 

1958 

Occupational Competency Tests , State Department of Education, Division of 
Vocational Education, Columbus, 0. 1962 

Panitz, Adolf and Olivo, C. Thomas, The State of the Art of Occupational 

Competency Testing , Rutgers University, Graduate School of Education, 
Department of Vocational/Technical Education, New Brunswick, N. J. 1970 

Popham, James, Performance Tests of Instructors Competence for Trade 6c 

Industrial Education , Final Report, University of California, Los Angeles, 
Calif. 1968 



Remmers, H. H. , Gage, U. L. and Rumrael, J. Francis, A Practical Introduction 
to Measurement and Evaluation , Harper and Row, New York, N. Y. 1965 

Segal, David, State Testing and Evaluation Programs , Federal Security Agency, 
Circular No. 320, Washington, D. C. 1951 



Shimberg, Benjamin, The Performance Phase of Trade Competency Examination , 

Unpublished paper. Educational Testing Service, Princeton, N. J. 1966 




169 






150 



Siegel, Arthur, and Courtney, Douglas, Development of Practical Performance 
Measures , (Under contract with the Office of Naval Research) Institute 
for Research in Human Relations, Philadelphia, Pa. 1953 

Stolurow, Lawrence M. , The Taxonomy of Learning Task Characteristics , Techni- 
cal Documentary Report AMRL-TDR- 64- 2 , Behavioral Sciences Laboratory, 
Wright Patterson Air Force Base, Ohio 1964 

Stuitt„ Dewey (Ed.), Personnel Research and Test Development in the (U.S.) 

Bureau of Naval Personnel , Princeton University Press, Princeton, N. J. 
1947 



Super, D. E. , Appraising Vocational Fitness by Means of Psychological Tests , 
Harper, New York, N. Y. 1949 

Swanson, R. A. , Auditory Automotive Mechanics Diagnostic Achievement Tests , 

(Unpublished doctoral dissertation) University of Illinois, Chicago, 111. 
1968 

The Ohio State University, Instructions to Test Administrators , Instructions 
Materials Laboratory, Trade and Industrial Education Science, State 
Department of Education, Division of Vocational Education, Columbus, 0. 

1965 

The University of the State of New York, Improving the Class Room Test , The 
State Education Department, Bureau of Test Development, Albany, N. Y. 

1964 

Thompson, Lor in Andrew and others, Interview Aids and Trade Questions for 
Employment Offices , Harper and Brothers, New York, N. Y. 1936 

Thorndike, Robert L. and Hagen, Elizabeth, Measurement and Evaluation in 
Psychology and Education , John Wiley & Sons, New York, N # Y. 1961 

Tinkelman, Sherman, Difficulty Prediction of Test Items , Bureau of Publica- 
tions, Teachers College, Columbia University, New York, N. Y. 1947 

U. S. Army Enlisted Evaluation Center, Performance Testing , Ft. Benjamin 
Harrison, Indianapolis, Ind. 1969 

U. S. Department of Labor > Occupational Licensing and the Supply of Non- 

Professional Manpower , Manpower Administration, Manpower Re'search Mono- 
graph #1., Superintendent of Documents, U. S. Government Printing Office, 
Washington, D . C. 1969 

U. S. Army Signal School, Performance Test Construction, Ft. Monmouth, N. J. 

1960 

U. S. Na^y, Procedures for Constructing and Evaluating Achievement Examinations , 
Unpublished Paper, Service School Command, U. S. Naval Training Center, 
Great Lakes, 111. 

U. S. Office of Education, Teacher Competency in Trade and Industrial Educa- 
tion , U. S. Department of Health Education & Welfare, OE 84006, 

Washington, D. C. 1960 



0 





1960 



Wood, D. A., Test Construction , Charles Merril Books, Columbus, 0. 

Wipple, James E. , Baldwin, Robert D. , Mager, F. and Vinsberg, Robert, 

A Threa Hour Performance Test to Evaluate Job Effectiveness of Army 
Radar Mechanics , Technical Report 38 Human P.esources Research Office, 
Alexandria, Va. 1965 

Walker, H. M. and Lew, J., Statistical Inference , Holt, Rinehart & Winston, 
New York, N. Y. 1953 



171 

o ' f 

ERLC 



152 



